This study investigates the nature and the validity of the PhonePass SET-10 test designed to measure test-takers’ English oral proficiency using the automated computer technology. For this study, the data from 84 Korean college students were collected: students’ TEPS scores, PhonePass SET-10 results as well as the results from the survey developed to measure their attitudes toward the new format of English speaking test. Based on the analysis of the study, it is found out that there exists a positive correlation between the TEPS scores and the PhonePass SET-10 results of the participants, indicating that in fact the computer-based automated evaluating system can significantly contribute to assessing students’ English oral proficiency while the items and the purpose of the test administration are relatively limited. It is also reported that the participants of this study showed positive attitudes toward the PhonePass SET-10 with a hope that the newly developed speaking test will help them further their English study.