Using a Rule-Based System to Identify Flawed Multiple-Choice Items
Tahereh Firoozi, Lia M. Daniels, Vijay John Daniels, Mark J. Gierl · 2025
Daniels et al. (2025) created use-inspired item-writing guidelines that map onto the processes and practices used by non-professional item writers in order to describe a logical sequence of steps that are used when creating new multiple-choice test items. But using guidelines in order to identify item-writing flaws is a time-consuming and laborious process because it requires human reviewers to examine the frequency of each specific flaw. To address this shortcoming we describe, evaluate, and demonstrate how a rule-based computer system can be used to implement item-writing guidelines in order to efficiently and accurately evaluate the quality of multiple-choice test items. The rule-based system was evaluated by comparing the ratings from a human reviewer with the rule-based system on a sample of test items resulting in almost perfect agreement demonstrating that the system can identify common multiple-choice item flaws. The system was also used to identify the item flaws found in a large sample of 1078 operational test items written in the content areas of research methods, statistics, child development, and educational psychology. The most common flaw was the logical ordering of options followed by grammar and spelling errors. Directions for future research are discussed.