Using Calibration Questions to Obtain Absolute Scaling in MaxDiff
Bryan K. Orme · 2009
In this paper, we create an artificial situation that demonstrates the relative scaling issue for MaxDiff at its worst. We collect a first wave of MaxDiff data on 30 items, and based on the items’ average scores we separate them into the best 15 and the worst 15 items. Then, we give two new sets of respondents MaxDiff questionnaires that include either the best 15 or the worst 15 items as determined from Wave 1 (plus a few calibration questions). With the addition of the calibration questions, we attempt to recover the original scaling of the first 30 items using only the data from wave 2. The two types of calibration questions involve a subset of five MaxDiff items judged using a 5-point rating scale, or the method of paired comparisons (MPC) involving best and worst items volunteered by the respondents (via open-end questions) versus a subset of five items included in MaxDiff. The 5-point scale calibration data allow us to rescale Wave 2 data and fit the original scaling of Wave 1 scores with R-Squared of up to 0.83. We find even greater success in a quali-quantitative calibration approach. A set of paired comparisons 1 The author thanks Rich Johnson and Lynd Bacon for their critique of earlier drafts. The opinions and any errors