An Efficient Non-parametric Item Selection Method for Polytomous Scoring CD-CAT
Junjie Li, Huijing Zheng, Chunhua Kang, Pingfei Zeng · Chinese/English Journal of Educational Measurement and Evaluation · 2023
In educational evaluations at home and abroad, polytomous scoring items are becoming increasingly important. They can provide richer and more valuable information, with unmatched advantages compared to binary (0-1 scoring) items. If used as a tool for teachers to diagnose and assess students in the classroom, CD-CAT (Cognitive Diagnostic Computerized Adaptive Testing) has significant implications for improving teaching effectiveness. However, in classroom teaching situations, it is not feasible to estimate item parameters accurately with a large sample, as in large-scale assessments. In such cases, non-parametric CD-CAT becomes the only viable option. Compared to parametric CD-CAT, non-parametric CD-CAT started later and is particularly lacking in research related to polytomous scoring. Item selection method is at the core of CD-CAT, so it is essential to develop a non-parametric item selection method suitable for polytomous scoring CD-CAT. This study proposes a non-parametric item selection method for polytomous scoring cognitive diagnostic computerized adaptive testing (PCD-CAT): the Manhattan Distance Non-parametric Difference index item selection method (MD-NDI). The results of simulation studies indicate: (1) MD-NDI item selection method is suitable for PCD-CAT scenarios and exhibits better performance when the item bank quality is poor or the sample size for estimating item parameters is limited. (2) MD-NDI does not require pre-testing of items and distributes item usage more evenly, effectively ensuring the security of the item bank. (3) Even in cases of incorrectly specified item bank of Qc-matrix, MD-NDI still shows higher pattern correct classification rates. (4) In the study of variable-length PCD-CAT, MD-NDI not only reduces the test length in most conditions but also has a higher pattern correct classification rates when reaching the test termination rule.