ExASAG: Explainable Framework for Automatic Short Answer Grading
Maximilian Tornqvist, Mosleh Mahamud, Erick Mendez Guzman, Alexandra Farazouli · 2023
As in other NLP tasks, Automatic Short Answer Grading (ASAG) systems have evolved from using rule-based and interpretable machine learning models to utilizing deep learning architectures to boost accuracy.Since proper feedback is critical to student assessment, explainability will be crucial for deploying ASAG in real-world applications.This paper proposes a framework to generate explainable outcomes for assessing question-answer pairs of a Data Mining course in a binary manner.Our framework utilizes a fine-tuned Transformer-based classifier and an explainability module using SHAP or Integrated Gradients to generate language explanations for each prediction.We assess the outcome of our framework by calculating accuracy-based metrics for classification performance.Furthermore, we evaluate the quality of the explanations by measuring their agreement with human-annotated justifications using Intersection-Over-Union at a token level to derive a plausibility score.Despite the relatively limited sample, results show that our framework derives explanations that are, to some degree, aligned with domain-expert judgment.Furthermore, both explainability methods perform similarly in their agreement with human-annotated explanations.A natural progression of our work is to analyze the use of our explainable ASAG framework on a larger sample to determine the feasibility of implementing a pilot study in a real-world setting.