A Data-Based Simulation Study of Reliability for an Adaptive Assessment Based on Knowledge Space Theory
Christopher W. Doble, Jeffrey Matayoshi, Eric Cosyn, Hasan Uzun, Arash Karami · International Journal of Artificial Intelligence in Education · 2019
A large-scale simulation study of the assessment effectiveness of a particular instantiation of knowledge space theory is described. In this study, data from more than 700,000 actual assessments in mathematics using the ALEKS (Assessment and LEarning in Knowledge Spaces) software were used to determine response probabilities for the same number of simulated assessments, for the purpose of examining reliability. Several measures of reliability were examined, borrowed from existing psychometric approaches, with an eye toward developing measures for evaluating reliability for adaptive assessments. The results are compared to analogous results for assessments having mathematics content overlapping that of the ALEKS assessment, and some consequences and future directions are discussed.