Face-to-face interview format has been widely used to elicit English language samples in oral proficiency testing. It is desirable to replicate target language use situations in oral testing, and the direct testing format is believed to assess more authentic and interactive language abilities. However, it has been argued that speech samples from unstructured face-to-face interviews are quite different from those in natural communication settings. The purpose of this study was to understand the construct to be assessed through an unstructured face-to-face interview, which was adopted to an English speaking contest. Data from the English speaking contest were analyzed in terms of how they were different from the characteristics of naturally occurring conversations. It was found that the test construct in the English speaking contest did not reflect the features of natural conversation specifically in the areas of turn-taking, adjacency pairs, and topic nomination due to asymmetrical power relations between interviewer and interviewees. Therefore, it was suggested that we need to incorporate diverse discourse-based approaches into current speaking skill assessments, which can interpret spoken language data in many ways.