Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
Menna Fateen, Bo Wang, Tsunenori Mine · IEEE Access · 2024
Automatic short answer scoring (ASAS) helps reduce the grading burden on educators but often lacks detailed, explainable feedback. Existing methods in ASAS with feedback (ASAS-F) rely on fine-tuning language models with limited datasets, which is resource-intensive and struggles to generalize across contexts. Recent approaches using large language models (LLMs) have focused on scoring without extensive fine-tuning. However, they often rely heavily on prompt engineering and either fail to generate elaborated feedback or do not adequately evaluate it. In this paper, we propose a modular retrieval augmented generation (RAG) based ASAS-F system, utilizing RAG as a few-shot selection method to score answers and generate feedback in zero-shot and few-shot learning scenarios. We design our system to be adaptable without extensive prompt engineering using an automatic prompt generation framework. Results show an improvement in scoring accuracy by 9% on unseen questions compared to fine-tuning, offering a scalable and cost-effective solution.