An Investigation of Knowledge-Based AI vs. Human Evaluation in the Context of Academic Summary Evaluation: Similarities, Dissimilarities, and Being Toward Mutual Understandings
Jinho Kim, Golnoush Haddadian, Min Kyu Kim · Proceedings. · 2023
This study aims to explore the similarities and dissimilarities of knowledge-based AI evaluations vs. human evaluations and discuss how they can be utilized for formative feedback in academic summary writing.Data were collected from 62 students who utilized AI-based formative feedback to make revisions to their summaries.We compared indices on three dimensions (surface, structure, semantic) that were automatically generated through this software with human-rated evaluations.MANOVA results of learners' initial draft and final revision showed learning gains in the semantic dimension and human-evaluated scores.Some significant correlations were observed between automatic and human-rated evaluations.Given that each measure can be interpreted to provide different insights, we suggest combining knowledge-based AI and human evaluations for rich and informative feedback.