Reliability Generalization: An HLM Approach
Jianjun Wang · Journal of instructional psychology · 2002
Hierarchical data structures have been identified in educational and psychological measurement, and statistical approaches are developed to partition the score variances at multiple levels of the data hierarchy. On basis of the classical test theory and the current HLM literature, the index has been constructed for unconditional and conditional models to facilitate generalization of the computing in various test settings. ********** is an important index in educational and psychological measurement. According to a joint committee of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education (1985), Reliability refers to the degree to which test scores are free from errors of (p. 19). the decrease of measurement errors is often associated with an increase of the measurement consistency in various circumstances, generalization may provide an important tool for characterizing score equality (Vacha-Haase, 1998, p. 16). In this study, the purpose is to discuss conditional and unconditional hierarchical models accounting for measurement errors at different levels that are essential to the generalization of reliability. the test score depends on many conditions of the test and subjects, empirical factors need to be introduced at the multiple levels to describe these conditions and facilitate generalization of the computing in different settings. Literature Review The classical test theory represents one of the cornerstones in educational and psychological measurement (Lord & Novick, 1968; Pedhazur & Schmelkin, 1991). Pedhazur and Schmelkin (1991) recollected: Since it was proposed by Spearman (1904), the tree-score model, or what has come to be known as classical test theory, has been the dominant theory guiding estimation of reliability (p. 83). Specifically, Novick, Jackson, and Thayer (1971) elaborated, In the classical test theory model, the observed score X on a person is taken to have expectation x, the true score for that person. The error score is defined by e = x - [tau]. The corresponding random variables defined over persons are related by the equation (1.1) X = T + E with [epsilon] (E|[tau]) = 0 (p. 261) Regarding the computing, Novick, Jackson, and Thayer (1971) added, The (intraclass correlation) of a test is defined as (1.3) [[rho].sup.2.sub.XT] = [[sigma].sup.2.sub.T]/[[sigma].sup.2.sub.x] = [[sigma].sup.2.sub.T]/([[sigma].sup.2.sub.T] + [[sigma].sup.2.sub.E]) = [[rho].sub.XX.sup.1] where X and X' are parallel measurements. (p. 261) In a test containing multiple items, student responses to each test item can be treated as an indicator of the true score. Thus, the responses to a set of items comprise multiple indicators of the individual performance. The hierarchical data structure is illustrated by the fact that the item responses are nested within each student. In addition, factors at the student level can be employed to reflect different test conditions, such as the differences in student demographics, past experiences, as well as the instructional coverage of the test contents. Hence, considerations of the multilevel factors are essential to a proper generalization of the assessment in various learning and/or testing environments. Vacha-Hasse (1998) searched the PsycINFO database for articles published from 1984 to July 1997, and conducted a meta-analysis on issues of generalization. She noted, Of the articles reviewed for the present study, 65.76% made absolutely no reference to reliability. At the other extreme, authors of only 13.06% of the articles reported coefficients for the data analyzed in the respective studies. …