Comparison of Prediction Performance of Computer Adaptive Testing with Machine Learning Methods in Response Patterns Suitable for Rasch Model with Simulation Study: A Methodological Research
Emrah Gökay Özgür, Beyza Doğanay Erdoğan · Turkiye Klinikleri Journal of Biostatistics · 2022
Objective: The scales measuring latent variables are used to gain information about the characteristic () levels of individuals. Scales can be investigated with classical methods as well as the Computer Adaptive Testing (CAT) method. In this study, the performance of machine learning algorithms, including Classification and Regression Tree (CART), Random Forest (RF), Gradient Boosting Machines (GBM) and Extreme Gradient Boosting Machines (XGBoost), were tested as a new approach to a CAT application with the algorithm routinely used in CAT in simulation data derived from different scenarios from the Rasch model. Material and Methods: In the CAT application, the Rasch model was used as the probabilistic model and the question selection based on the information criterion was used as the question selection criterion. The performances of the methods were compared, based on the average number of items, the square root of the mean squared error (RMSE), the intraclass correlation coefficient (ICC) and the average prediction value obtained from these predictions. Results: Different methods become superior to others as category numbers increased. When the number of items was considered, the CART method made good predictions with the least number of items. When RMSE values were analysed, both GBM and XGBoost methods had low RMSE values. The methods compared have a good ICC value in estimating total scores. Conclusion: As a result of general comparisons, we recommend that a person planning a new study use machine learning methods which are frequently used in different fields recently as an alternative to the CAT method in accordance with its purpose.