Automatic conversational assessment using large language model technology
Jan Bergerhoff, Johannes E. Bendler, Stefan A. Stefanov, Enrico Cavinato, Leonard Esser, Tommy Tran, Aki Härmä · 2024
Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.