On how testwiseness and acceptance reluctance influence the validity of sequential knowledge tests
Martin Papenberg · Univ. Duesseldorf: Duesseldorfer Dokumenten- und Publikationsserver · 2018
Multiple-choice tests are widely used to assess cognitive ability, scholastic achievement and knowledge. A multiple-choice (MC) item consists of a question stem that is shown along with the correct answer and one or several distractors. From among these response options, test-takers have to identify the solution. Test scores derived from MC tests have been shown to satisfy high psychometric standards if guidelines for good item writing are followed. However, investigations of MC tests in practice revealed a high prevalence of flawed item writing. By comparing the response options that are shown simultaneously, testwise test-takers can capitalize on such flaws to increase their test scores, making MC tests biased against less testwise test-takers. As a consequence, MC test scores may contain construct-irrelevant variance due to individual differences in testwiseness. Discrete-option multiple-choice (DOMC) testing has been proposed as a means to reduce such construct-irrelevant variance. In DOMC items, the question stem is presented together with only one response option at a time; additional options are shown only after the correctness of the present option has been assessed in a yes/no decision. Thus, response options are presented sequentially rather than simultaneously, precluding the possibility to compare the relative plausibility of all alternatives. Previous investigations indicated that DOMC testing makes items more difficult, reduces the number of response options that have to be shown, and reduces testing time as compared to traditional MC testing. The present thesis expands the yet small body of research on the DOMC test format. First, it is shown that DOMC is capable of reducing the effectiveness of testwiseness strategies, as indicated by the observation that the gap between DOMC and MC test scores increases as the level of testwiseness increases. Next, by experimentally establishing different levels of knowledge, we find that MC and DOMC test scores are equally capable of representing known ability levels, suggesting that MC and DOMC test scores are equally valid. In exploratory analyses, we also find that DOMC test-takers very rarely select the correct answer when they have no knowledge. This gives surprising evidence to the notion that DOMC testing reduces test-takers’ chances to randomly guess a solution. The remaining part of this thesis proposes and investigates a new model of response behavior in sequential tests (MORBIST). On the basis of signal detection theory, MORBIST describes the processes that lead to DOMC item responses. Simulations based on MORBIST predict that DOMC test scores do not only depend on test-takers’ ability, but also on their knowledge-independent individual preference to choose late rather than early response options in DOMC items. We first demonstrate the existence of such individual differences in acceptance reluctance in a correlational investigation. Acceptance reluctance is thereby identified as an important cognitive trait in the domain of sequential knowledge tests. Consistent with MORBIST’s prediction, the results of the correlational study also show that higher acceptance reluctance is related to better test scores in a DOMC knowledge test. Additional evidence of MORBIST’s usefulness is obtained by reproducing several known properties of the DOMC test format in a computer simulation. In a concluding experimental study, we establish low versus high acceptance reluctance using a payoff manipulation. The results confirm that high acceptance reluctance causally leads to increased DOMC test scores. Another computer simulation also shows that MORBIST can adequately reproduce the observed test score gap between high and low acceptance reluctance. Taken together, the present research shows first evidence to the notion that DOMC test scores contain some construct-irrelevant variance. To obtain an integrated judgment on the viability of DOMC testing, this drawback has to be weighed up against the advantages of the DOMC test format that include a reduced susceptibility to testwiseness and random guessing.