Stepping into the Future
Loren J. Naidoo, Deone Zell · 2024
The arrival of generative artificial intelligence (AI) offers new opportunities to use AI to help instructors grade written work, considered an important but time-consuming task. This paper investigates both the interrater and test-retest reliability of human and generative AI grading. Results of interrater reliability analyses suggest that AI can grade written answers as reliably as humans and in less time, although it is still prone to errors. Results of test-retest reliability analyses suggest that AI can be at least as reliable as human graders. Together, these findings suggest that generative AI may be a useful tool for educators and should continue to be studied. However, in our view human review is still needed to understand the mechanisms being used (e.g., keyword search versus holistic) and ensure AI&s;s grading accuracy.