Investigating hybrid test designs in passage-based adaptive tests
Ye Cheryl, Stephen B. Dunbar, Deborah J. Harris, Robert D. Ankenmann, Sanvesh Srivastava, Catherine J. Welch, Terry A. Ackerman · 2020
Computerized adaptive testing (CAT) selects each item to match with an examinee’s ability level during the test. In contrast, multistage adaptive testing (MST) administers a group of items instead of each individual item adaptively. A hybrid design combines CAT and MST borrowing strengths from both testing modes. However, existing research studies indicate that there is lack of knowledge on hybrid designs in passage-based adaptive testing, including situations where misrouting occurs. Misrouting refers to when examinees get routed to paths that don’t match with their true ability levels. Therefore, the purpose of this study is to evaluate the proposed hybrid designs (HMCAT designs) with respect to ability estimation accuracy under different MST configurations and their performance when misrouting occurs. The HMCAT designs under passage-based adaptive testing control test adaptability at both passage/stimulus level and item level. Four proposed HMCAT designs differ by implementing the item level CAT at different stages of the MST. Specifically, among all four designs, the two most extreme designs are the one implementing CAT across all stages (CC) and the one not implementing CAT across any stage (PP). The two hybrid designs include implementing CAT only in the routing stage (CP) and implementing CAT only in the last stage (PC). The hybrid designs’ performance compared to that of the CC design and the PP design when interacting with different MST configurations and when misrouting occurs are the two main research questions of the study. The three-phase simulation analysis indicates that (1) the PC design achieves more accurate final ability estimation results under the three-stage and the four-stage MST configurations; (2) although the CP design fails to perform effectively when misrouting occurs under the three-stage MST configuration, it is still able to achieve accurate final ability estimation results under both two-stage MST configurations; (3) the CC design and the PP design achieve similar and the most accurate final ability estimation results under all four MST configurations; (4) the PC design reaches similar accuracy as the CC design and the PP design under the three-stage and the four-stage MST configurations; (5) the CP design reaches a similar accuracy level as the CC design and the PP design under both two-stage MST configurations; (6) the CP design is affected the most when misrouting occurs in the three-stage MST configuration, whereas the other three HMCAT designs are not affected. In general, findings from simulation analyses suggest that the HMCAT designs are effective enough to use in passage/stimulus-based adaptive tests with appropriate MST configurations. They also contain advantages such as greater flexibility with respect to content balancing, test security and control on item administration. However, it is important to understand that the performance of HMCAT designs can heavily depend on the MST configuration, the item pool characteristics, the test design and components in the test administration algorithm, such as content balancing strategies. Results from the current study provide implications for practitioners on how to decide on an appropriate HMCAT design under real testing contexts and how to evaluate and maintain the selected HMCAT design to effectively implement the design in practice.