A validity-based approach to quality control and assurance of automated scoring
Isaac I. Bejar · Assessment in Education Principles Policy and Practice · 2011
Automated scoring of constructed responses is already operational in several testing programmes. However, as the methodology matures and the demand for the utilisation of constructed responses increases, the volume of automated scoring is likely to increase at a fast pace. Quality assurance and control of the scoring process will likely be more prominent as a result. A validity-based approach to ensure the quality of scores based on automated means is proposed. First, an argument is made for aligning quality assurance/control with validity, rather than viewing them as separate processes. Second, to pre-emptively avoid quality and design defects that would erode the validity of scores, it is argued that quality must be ‘designed in’ through the assessment design process by attending to the interdependencies among the different components of an assessment, including scoring. Third, key elements of the design of scoring engines, evidence extraction and evidence synthesis, are discussed as a further avenue for maintaining score quality.