"Types" of Classification Consistency from the Perspective of Replications
Hyun Sook Yi · Journal of Curriculum and Evaluation · 2007
Classification consistency is often considered as reliability of a measurement procedure involving classifications of examinees into a set of ordered categories. In order to quantify classification consistency, an investigator should define the universe of intended replications, and determine data collection designs as well as methods for estimating classification consistency that would be the most suitable for the intended replications. This process is important because classification consistency estimated from different conceptualizations about replications may produce different types of classification consistency estimates. This paper differentiates three types of classification consistency from the perspective of replications (test-retest classification consistency; alternate-forms classification consistency; and internal classification consistency), and introduces three model-based approaches to estimating classification consistency based on each conceptualization, with focuses being given to the intended universe of replications. An empirical analysis will be provided at the end to illustrate differences in estimation according to different conceptualizations about the universe of replications.