Managing the Intricacies of Teaching Evaluation Data with Mixture Cross-Classified Item Response Theory Models
R. Maximilian Bee, Tobias Koch · 2023
The article introduces models for cross-classified multirater data, where multiple raters rate multiple targets on items with ordered response categories. Moreover, it is assumed that the rater population is heterogeneous and consists of latent subpopulations (mixture distribution assumption). The resulting models are mixture cross-classified item response theory models. Two version were developed: a graded response model and a generalized partial credit model. All models were defined on the principles of stochastic measurement theory and fit via Bayesian estimation by means of the open-source software Stan.For model assessment and comparison, leave-one-out cross-validation as well as stacking was used. The model code is available as Supplementary Material and contains an integration algorithm to calculate the marginalized likelihood that can be easily adapted to similar models. The analysis of teaching evaluation data reveals two latent rater populations that show extreme vs. moderate response styles. These results suggest that the heterogeneity of the student population in teaching evaluation research should be considered. The results of a simulation study show that the parameters can be satisfactorily recovered in the developed models.