Machine-assisted writing evaluation: exploring pre-trained language models in analyzing argumentative moves
Wenjuan Qin, Weiran Wang, Yuming Yang, Tao Gui · Computer Assisted Language Learning · 2025
The study investigates the efficacy of pre-trained language models (PLMs) in analyzing argumentative moves in a longitudinal learner corpus. Argumentative writing requires a range of complex cognitive skills, including logical reasoning, evidence support, and acknowledgment of opposing positions. Prior studies on argumentative moves often rely on qualitative analysis and manual coding, limiting their efficiency and generalizability. The study aims to: (1) assess the reliability of PLMs in analyzing argumentative moves; (2) utilize PLM-generated annotations to illustrate developmental patterns and predict writing quality. A longitudinal corpus of 1,643 argumentative texts from 235 English learners in China is collected and annotated into six move types: claim, data, counter-claim, counter-data, rebuttal, and non-argument. The corpus is divided into training, validation, and application sets annotated by human experts and PLMs. We use Bidirectional Encoder Representations from Transformers (BERT) as one of the implementations of PLMs. The results indicate robust reliability of PLMs in analyzing argumentative moves, with an overall F1 score of 0.743, surpassing existing models in the field. Additionally, PLM-labeled argumentative moves effectively capture developmental patterns and predict writing quality. Over time, students exhibit an increase in the use of data and counter-claims and a decrease in non-argument moves. While low-quality texts are characterized by a predominant use of claims and data supporting only one-sided positions, mid- and high-quality texts demonstrate an integrative perspective with a higher ratio of counter-claims, counter-data, and rebuttals. This study underscores the transformative potential of integrating artificial intelligence into language education, enhancing the efficiency and accuracy of evaluating students’ writing. The successful application of PLMs can catalyze the development of educational technology, promoting a more data-driven and personalized learning environment that supports diverse educational needs.