Quantifying Computational Thinking Skills: an Exploratory Study on Bebras Tasks
Ana Liz Souto Oliveira, Wilkerson L. Andrade, Dalton Serey, Monilly Ramos Araujo Melo · Journal of the Brazilian Computer Society · 2025
Computational Thinking (CT) is a cognitive problem-solving approach commonly employed in the field of Computer Science. Over recent years, various strategies have emerged to promote CT awareness and understanding. Despite these initiatives, there has been a lack of quantitative analysis aimed at assessing CT as a cognitive skill among undergraduate students, particularly focusing on items designed for this purpose. In this study, our objective is to investigate the psychometric properties of CT questions as answered by novice Computer Science undergraduates. To achieve this, we selected a set of questions from the Bebras Challenge, an international competition designed to explore CT skills without requiring programming expertise. In pursuit of our goal, we utilized Item Response Theory (IRT) to scrutinize the difficulty and discrimination levels of these selected Bebras questions. Difficulty is related to how an examinee responds to an item, while discrimination measures how effectively an item can differentiate between individuals with higher and lower levels of knowledge. Our findings reveal several key insights: (i) Concerning the accuracy in predicting question difficulty, theoretical predictions achieved an accuracy rate ranging from 53% to 58% when compared to empirical data. (ii) The Bebras Challenge questions predominantly exhibited two levels of difficulty, spanning from easy to medium. (iii) The questions displayed a spectrum of discrimination levels, encompassing low, moderate, and high discrimination, a crucial aspect for crafting effective assessment instruments. Additionally, we have gathered observed lessons from this exploratory study, regarding the design of questions that can contribute to reliably measure CT skills. These lessons contribute to understanding features influencing the reliable design of items for measuring CT skills. These insights serve as a resource for future research endeavors aimed at enhancing our understanding of assessing CT abilities.