Adaptive Learning Curve Analytics with LLM-KC Identifiers for Knowledge Component Refinement
Jing Fan, Tsvetomila Mihaylova, Bita Akram, Narges Norouzi, Peter L. Brusilovsky, Arto Hellas, Juho Leinonen · 2025
Accurate modeling of student knowledge is essential for delivering timely, targeted feedback in Intelligent Tutoring Systems (ITS). Knowledge Components (KCs)—discrete units of domain knowledge—have traditionally been handcrafted by experts, a process that is both time-consuming and difficult to scale. In this work, we replicate a recent Large Language Model (LLM)-based approach to automate KC extraction called LLM-KC Identifier (LLM-KCI) and extend on it by evaluating the extracted KCs using learning curve analysis. By comparing LLM-generated KCs against expert-annotated counterparts in an introductory programming course, we demonstrate that LLMs not only match experts in capturing core concepts but also bring unique advantages: consistent identification across diverse assignments and scalability. Through static overlap metrics (Jaccard similarity, overlap coefficient) and learning curve analyses, we show that certain LLMs (e.g., GPT-4o, DeepSeekR1) produce error-rate trajectories as smooth or smoother than expert-annotated KC models, effectively isolating student learning trends. Our findings suggest that automated KC extraction can become a mainstream tool for personalized learning analytics, enabling educators to rapidly adapt curriculum and interventions at scale.