This study investigated the effect of automated writing evaluation (AWE) feedback on Korean university students’ revision behavior. Specifically, it explored the extent to which the quality of essays differed between first drafts and second drafts. Furthermore, the way individual students dealt with incorrect feedback in revising essays was examined. In this study, ETS’ Criterion program, one of the most popular AWE programs, was employed. In cases where incorrect feedback was detected, the first and second drafts were closely compared to find out how students altered the text in response to the incorrect feedback, and this was classified into one of three categories: successful changes, unsuccessful changes, and text deletion. The results showed that students’ second drafts were rated significantly higher than first drafts by two NS raters, which indicates that many students followed the revision suggestions made by the Criterion program. As students’ Criterion scores increased, their ability to identify and make successful changes in response to incorrect feedback improved. The findings of the study can advance our understanding of AWE use in an EFL context and should contribute to broader examination of how Korean university students engage in revision of their essays.