Validity Argument Roadmap for Automated Scoring
David W. Dorsey, Hillary R. Michaels, Steve Ferrara · 2024
This chapter offers a validity argument roadmap for automated scoring of essays and short, constructed response items that include writing and speaking. It begins by providing a focused review of the relevant Standards for Educational and Psychological Testing and then highlights current principles and practices of artificial intelligence (AI)-enabled scoring, as they relate to validity arguments. These include how AI scoring engines are optimized for specific scoring tasks, the meaning of “fairness” and its place in validity arguments, and important changes in the AI landscape such as explainable and responsible AI. Finally, attention is turned to the call for more principled approaches to constructing validity argument technical reports. The issues raised above are summarized in the form of a practical roadmap that practitioners and researchers alike can use to construct better validity arguments involving AI-enabled scoring. The chapter concludes with a consideration of ongoing challenges and areas for future research.