Reflective prompt engineering: a new strategy for automated short answer scoring in biology
Moriah Ariely, Asaf Salman, Anat Yarden, Giora Alexandron · International Journal of Science Education · 2025
The educational landscape is rapidly evolving, driven by Generative Artificial Intelligence (GenAI) advancements. As GenAI transforms educational practices, a key area of impact is developing automatic scoring systems to efficiently and accurately assess students’ problem-solving and explanation skills. This study introduces a novel prompt engineering strategy, Reflective Prompt Engineering (RPE), which applies iterative human-AI collaboration through discussion and reflection with a powerful LLM to enhance scoring performance. In RPE, human experts guide the AI by integrating its inferred criteria and language into subsequent prompts, enabling reflective alignment and improvement. We applied RPE to score open-ended biology items using analytic grading rubrics, evaluating its performance against two benchmarks: a BERT-based scoring model and multiple examples prompts with no discussion. Performance (accuracy and Cohen’s Kappa) was assessed in two scenarios: within-item and cross-items. RPE achieved excellent agreement with human experts (Cohen’s Kappa > 0.8) using only 40–60 examples and consistently outperformed the multiple examples strategy, achieving significantly higher Kappa values. Our findings highlight the potential of RPE to optimise AI alignment with human-designed tasks and open pathways for broader applications across diverse domains, advancing automated evaluation practices with enhanced precision and adaptability.