Assessing Teamwork Skills: Can a Computer Algorithm Match Human Experts?
Igor Kotlyar, Tina Sharifi, Lisa Fıksenbaum · International Journal of Artificial Intelligence in Education · 2022
Teamwork skills are commonly evaluated by human assessors, which can be logistically challenging and resource intensive. Technological advancements provide an opportunity for a new assessment method – virtual behavioural simulations with self-scoring algorithms. This study explores whether a rule-based algorithm can match human assessors at evaluating teamwork skills. 206 undergraduate students completed a virtual simulation assessment, where they interacted with “teammates” (represented by chatbots) using natural language. In this study, students’ teamwork skills were assessed independently by a computer algorithm and two human experts based on the transcripts of their conversations with “teammates” (chatbots). The relative accuracy of these assessments was evaluated against peer- and self-evaluations of teamwork. The assessment scores generated by the algorithm and human experts were highly correlated with each other and were comparable in their ability to predict teamwork. The scores generated by the algorithm were slightly more correlated with peer-evaluations than those generated by human experts ( r = .25 and r = .17, respectively; p = .21). The results indicate that AI-based techniques offer a promising method of skill assessment to support learning and acquisition teamwork skills.