Calibrating CAT Pools and Online Pretest Items Using Marginal Maximum Likelihood Methods
Mary Pommerich, Daniel O. Segall · 2004
Research discussed in this paper was conducted as part of an ongoing large-scale simulation study to evaluate methods of calibrating pretest items for computerized adaptive testing (CAT) pools. The simulation was designed to mimic the operational CAT Armed Services Vocational Aptitude Battery (ASVAB) testing program, in which a single pretest item is embedded, or seeded, into the administration of an operational CAT. The overall goal of the research is to select a calibration method that will best represent the data and maintain a consistent scale over time, as new calibrations are conducted, pretest items are formed into new pools, and operational pools are replaced with the new pools. In this study, pretest and operational CAT items were simultaneously calibrated and placed on the scale of the operational parameters from one CAT pool that is designated as the anchor CAT pool. A primary objective of this simulation was to evaluate the performance of the Marginal Maximum Likelihood (MML) three-parameter logistic model based calibration methods when some of the items do not fit a 3PL model. The first method evaluated used Bilog-MG to fix parameters for anchor items to their known values and simultaneously estimate the parameters for nonanchor items. The second method used Bilog-MG to estimate parameters for nonanchor items only and then used MML transformation procedures to rescale those parameters to the scale of the anchor items. Results suggest that it is safer to calibrate the nonanchor items to a N(0,1) scale and then rescale to the scale of the anchor items rather than to fix the parameters for the anchor items and simultaneously calibrate the nonanchor items. (Contains 19 figures and 8 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document. PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY