So Happy Together? Combining Rasch and Item Response Theory Model Estimates with Support Vector Machines to Detect Test Fraud
Sarah Thomas · 2016
Each year, thousands of people must pass standardized exams to be certified in medical, legal, clinical, and technological fields.Unfortunately, the increasing number of examinees taking such tests seems to have been accompanied by an increase in the instances of reported cheating and the invention of more sophisticated cheating techniques.The stealing and sharing of proprietary test content, such as when items are recorded and then compromised via sharing, is associated with legal consequences for the culprit but can also have negative consequences for the integrity, reputation, and budget of a testing program.Compromised items represent a significant threat to the integrity of testing.The purpose of the present study was to detect items that were suspected to be compromised using a combination of Rasch and Item Response Theory model estimates, along with other item properties such as average response times and local dependence estimates, with Support Vector Machines (SVMs).In this study, we used this combination of methods to detect items of an international healthcare certification exam (N = 13,584) that were suspected to be compromised in screenshots or notes.The results showed that this method appeared to be somewhat accurate at classifying suspected compromised and suspected uncompromised items, but that the main factor driving these results appeared to be the relative size of the classes.The SVMs showed apparent bias towards predicting the items into the category of whichever item class was larger.Thus, the accuracy of the SVMs, balanced for class size, was much lower than desired.We hypothesized that the most important item features would be Rasch item infit and outfit, but the item feature results showed that the two most important features were the Rasch model standard error and the Rasch item difficulty.We also hypothesized that Rasch