AUTOMATIC SCORING OF ADVANCED PLACEMENT BIOLOGY ESSAYS
Jill C. Burstein, Randy M. Kaplan, Susanne Wolff, Chi Lu · ETS Research Report Series · 1997
ABSTRACT The Natural Language Processing Initiative (NLPI) at ETS currently does research and development for computer‐assisted scoring of natural language constructed‐responses. The purpose of NLPI's work is to develop computer‐assisted methods for scoring so that computer‐administered natural language constructed‐response items can be integrated into ETS' tests which can be scored efficiently with regard to time and cost. Until recently, NLPI's automated scoring efforts were primarily devoted to the development of computer programs used to score short‐answer responses constructed‐responses of up to 15 words. In previous studies, we have developed scoring systems for main idea item types (e.g., GUIDES items) and divergent thinking item types (e.g., the Formulating‐Hypotheses item (FH)). In the current study, we applied similar techniques to those used to score short‐answer responses in order to implement a prototype to automatically score AP Biology essays. This paper reports on the development process involved in building a linguistically‐based, computer scoring prototype for AP Biology essays and the results of the scoring system. The prototype was used to score essays that had been assigned an Excellent rating (9 or 10 points) or a Poor rating (0 ‐ 3 points) by human graders.1 The results reported in the paper illustrate agreement between human rater scores and computer‐based scores. The results of the study are promising and would encourage continued research in automated scoring of essays with highly analytical rubrics.