Surprisal as a Predictor of Essay Quality
Gaurav Kharkwal, Smaranda Muresan · 2014
Modern automated essay scoring systems rely on identifying linguistically-relevant features to estimate essay quality. This paper attempts to bridge work in psy-cholinguistics and natural language pro-cessing by proposing sentence process-ing complexity as a feature for automated essay scoring, in the context of English as a Foreign Language (EFL). To quan-tify processing complexity we used a psy-cholinguistic model called surprisal the-ory. First, we investigated whether es-says ’ average surprisal values decrease with EFL training. Preliminary results seem to support this idea. Second, we in-vestigated whether surprisal can be effec-tive as a predictor of essay quality. The results indicate an inverse correlation be-tween surprisal and essay scores. Overall, the results are promising and warrant fur-ther investigation on the usability of sur-prisal for essay scoring. 1