A Comparison Study of the Unidimensional IRT Estimation of Compensatory and Noncompensatory Multidimensional Item Response Data
Terry A. Ackerman · 1987
The purpose of this study was to compare the characteristics of unidimen sional ability estimates obtained from data generated from the multidimen sional IRT (MIRT) compensatory and noncompensatory models. Reckase, Carlson, Ackerman and Spray (1986) reported that when the compensatory model is used and item difficulty is confounded with dimensionality, the composition of the unidimensional ability estimates differs for different points along the unidimensional ability scale. Eight data sets (four compensatory, four noncompensatory) were generated for four different levels of correlated two dimensional abilities: p = 0, .3, .6, .9. In each set difficulty was con founded with dimensionality. Each set was then calibrated using the IRT calibration programs LOGIST and BILOG. BILOG calibration of response vectors generated to the matched MIRT item parameters appeared to be more affected than LOGIST by the confounding of difficulty and dimensionality. As the correlation between the generated two-dimensional abilities increased, the response data appeared to become more unidimensional as evidenced in bivariate plots of vs. 0 2 for specified 0 quantiles. A Comparison Study of the Unidimensional IRT Estimation of Compensatory and Noncompensatory Multidimensional Item Response Data One of the underlying assumptions of unidimensional item response theory (IRT) models is that a person's ability can be estimated in a unidimensional latent space. However, researchers and educators have expressed concern whether or not the response process to any one item requires only a single latent ability. Traub (1983) suggests that many cognitive variables are brought to the testing task and that the number used varies from person to person. Likewise, the combination of latent abilities required by individuals to obtain a correct response may vary from item to item. Caution over the application of unidimensional IRT estimation of multidimensional response data has been expressed by several researchers including Ansley and Forsyth (1985); Reckase, Carlson, Ackerman, and Spray (1986); and, Yen (1984). Using a compensatory multidimensional IRT (MIRT) model, Reckase et a l . (1986) demonstrated that when dimensionality and difficulty are confounded (i.e., easy items discriminate only on 0 L, difficult items discriminate only on 02) the unidimensional ability scale has a different meaning at different points on the scale. Specifically, for their twodimensional generated data set, upper ability deciles differed mainly on 02 while the lower deciles differed mostly on 0 1# These results led the authors to suggest that the univariate calibration of two-dimensional response data can be explained in terms of the interaction between the multidimensional test information and the distribution of the two-dimensional abilities. Reckase et al. (1986) examined the condition in which ability estimates were uncorrelated. Such an approach may not be very realistic, however, since most cognitive abilities tend to be