Capturing Collaborative Competency with GPT-4o and ENA
Yoonjae Lee, Christine Kwon, Sarah Seoh, Gahgene Gweon, John C. Stamper, Carolyn Penstein Rosé · Computer-supported collaborative learning/The Computer-Supported Collaborative Learning Conference · 2025
Collaboration is a critical learning skill employed across many educational domains.Since moments of collaborative competencies frequently occur in student discourse, there has been a growing body of work in automatic collaborative process analysis.With recent advancements in Large Language Models (LLMs), it became feasible to automate the collaborative process analysis through simple prompting methods.In this study, we develop CoComTag, an LLM-powered approach using GPT-4o that captures students' collaborative competency using four prompting techniques.Our findings demonstrate that CoComTag shows substantial agreement with humans (kappa of 0.67).Such a result shows that LLM yields comparable performance with prior studies using only prompting strategies without additional fine-tuning.Next, using qualitative error analysis and epistemic network analysis (ENA), we examine how CoComTag differs from humans in detecting collaborative competency.Our analysis shows that ENA provides visual cues on where collaborative competencies co-occur, revealing the differences between LLM and human annotation patterns.