On the Opportunities of Large Language Models for Programming Process Data
John Edwards, Arto Hellas, Juho Leinonen · 2025
Computing educators and researchers have long used programming process data to understand how students construct programs and address challenges.Despite its potential, fully automated feedback systems remain underexplored.The emergence of Large Language Models (LLMs) offers new opportunities for analyzing programming data and providing formative feedback.This study explores using LLMs to summarize programming processes and deliver formative feedback.A case study analyzed keystroke-level data from an introductory programming course, processed into code snapshots.Three state-of-the-art LLMs -Claude 3 Opus, GPT-4 Turbo, and LLaMa2 70B Chat -were evaluated for their feedback capabilities.Results show LLMs effectively provide tailored feedback, emphasizing incremental development, algorithmic planning, and code readability.Our findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.