Rating Short L2 Essays on the CEFR Scale with GPT-4
Kevin P. Yancey, Geoffrey T. LaFlair, Anthony Verardi, Jill C. Burstein · 2023
Essay scoring is a critical task used to evaluate second-language (L2) writing proficiency on high-stakes language assessments.While automated scoring approaches are mature and have been around for decades, human scoring is still considered the gold standard, despite its high costs and well-known issues such as human rater fatigue and bias.The recent introduction of large language models (LLMs) brings new opportunities for automated scoring.In this paper, we evaluate how well GPT-3.5 and GPT-4 can rate short essay responses written by L2 English learners on a high-stakes language assessment, computing inter-rater agreement with human ratings.Results show that when calibration examples are provided, GPT-4 can perform almost as well as modern Automatic Writing Evaluation (AWE) methods, but agreement with human ratings can vary depending on the test-taker's first language (L1).