This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters’ prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters’ prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment.

This paper first surveys three kinds of learner interview corpora (LINDSEI, NICT-JLE Corpus, and Trinity Lancaster Corpus), paying particular attention to their interview structures. Then, it explains the principles and features of the ICNALE Spoken Dialogue (ICNALE SD), which includes 425 videos and approximately 1.6-million-word transcripts of the L2 English interviews with 405 learners from ten regions in Asia and twenty native speakers. The ICNALE SD is one of the largest learner interview corpora and practically the sole dataset for the analysis of dialogue speeches by various Asian learners. As a case study based on the ICNALE SD, the author sought to find out how fluently learners in different regions speak in the interviews, which words they characteristically use, and which relationship is observed among them.