COMPUTER ANALYSIS OF ESSAY CONTENT FOR AUTOMATED SCORE PREDICTION: A PROTOTYPE AUTOMATED SCORING SYSTEM FOR GMAT ANALYTICAL WRITING ASSESSMENT ESSAYS
Jill C. Burstein, Lisa Braden‐Harder, Martin S. Chodorow, Shuyi Hua, Bruce Kaplan, Karen Kukich, Chi Lu, James J. Nolan, Don Rock, Susanne Wolff · ETS Research Report Series · 1998
ABSTRACT This report discusses the development and evaluation of a research prototype system designed to automatically score essay responses to the GMAT Analytical Writing Assessments: (a). Analysis of an Argument (Argument essays) and (b). Analysis of an Issue (Issue essays) item types. The system, Electronic Essay Rater ( e‐rater ), was designed to automatically analyze several features of an essay and score the essay based on the features of writing as specified in holistic rubrics. E‐rater uses a hybrid feature methodology. It incorporates several variables that are derived statistically, extracted through NLP techniques, or achieved by simple “counting” procedures. The version of the e‐rater described in this report uses five sets of critical feature variables to build the final linear regression model used for predicting scores. The same set of critical variables was used to fit models for the issue and argument training essays and the following results were achieved. For the set of 275 cross‐validation data, exact or adjacent agreement with human rater scores reached 95%. For the 282 cross‐validation issue essays exact or adjacent agreement with human rater scores achieved 93%. The rich feature variables used as score predictors in e‐rater could potentially be used to generate explanation of score predictions, and diagnostic and instructional information.