A comparison of various approaches to transitioning from linear tests to adaptive tests
Lida Chen, Terry Ackerman, Kate Cowles, Won‐Chan Lee, C. Stuart Welch, Stephen Dunbar · 2020
Throughout the past decades, many testing programs have completed or have started the transition from a linear testing format to an adaptive one. The frequently cited advantages of adaptive testing lie in the improvement of measurement efficiency by having the examinees take a shorter test while maintaining or increasing measurement accuracy at all ability levels. Among the many considerations that are involved regarding such a transition, the testing programs may be firstly interested in the appropriateness of the adaptive testing mode, given the purpose of the assessment. The purpose of this study was to compare three major adaptive testing modes, including the two more traditional and prevalent ones of computerized adaptive testing (CAT) and multistage testing (MST), and the more recently proposed multistage testing by shaping on the fly (MST-S), with respect to measurement accuracy and item pool usage. Under each adaptive testing mode, different designs and variations were compared, including the considerations of item selection methods in CAT, test assembly approaches and routing strategies in MST, and number of iterations in MST-S. Results suggested a promising future for MST-S, which has not been fully implemented in operational practice, in terms of its flexibility in balancing measurement accuracy and item exposure, over either CAT or MST. Although MST-S may be recommended in the specific context in this study, one should take the findings with caution in other applications under different contexts to avoid overgeneralization. This study adds to the literature of comparative studies on adaptive testing modes by including MST-S into the comparison in a very practical context. Also, the lesser-focused and understudied aspects of the traditional MST were discussed, especially those related to test assembly issues. Finally, the findings of this study provide reference for other testing programs who are interested in such a transition in a similar context.