LLM Generative AI and Students’ Exam Code Evaluation: Qualitative and Quantitative Analysis
Ema Smolić, M. Pavelić, Bartol Boras, Igor Mekterović, Tomislav Jagušt · 2024
Since the introduction of generative artificial intelligence (GAI) technology in the context of large language models (LLMs), it has been widely used for information extraction and/or extrapolation from different sources. In computer science education, a potential application of such technology is for automatic code review, i. e. shifting the burden of debugging non-compilable code, detecting overlooked optimization concerns such as poor memory management in code that otherwise passes automated tests, and other advanced tasks from a human grader to LLMs. However, LLMs are currently not capable of evaluating code or mathematical expressions with 100% reliability, i. e. beyond token pattern recognition and subsequent probabilistic answer generation. With that in mind, in this paper, we explore the risk of incorrect LLM code evaluation, both descriptive and numerical, as well as begin research on its mitigation and propose further work directions.