Automated Scoring with Validity in Mind
Isaac I. Bejar, Robert J. Mislevy, Mo Zhang · 2016
In this chapter we present an overview of automated scoring and illustrate some of the design decisions that should be considered when developing or evaluating an assessment that relies on automated scoring. The overarching theme is score validity. Given the increasingly ambitious goals of twenty-first-century assessments, it is important to approach their design in a systematic fashion to ensure that the scores that will eventually be reported are valid. The evidence-centered design (ECD) framework is especially well-suited for the task because it pays special attention to characterizing the evidence in performances. We focus on the evidence identification component of an assessment design within this approach, which identifies and implements procedures for extracting evidence of the proficiencies an assessment claims to support. We illustrate key design decisions in a wide range of domains from writing assessment to games and simulations.