Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks
Yvette Oortwijn, Thijs Ossenkoppele, Arianna Betti · UvA-DARE (University of Amsterdam) · 2021
We present a systematic procedure for interrater disagreement resolution. The procedure is general, but of particular use in multiple-annotator tasks geared towards ground truth construction. We motivate our proposal by arguing that, barring cases in which the researchers’ goal is to elicit different viewpoints, interrater disagreement is a sign of poor quality in the design or the description of a task. Consensus among annotators, we maintain, should be striven for, through a systematic procedure for disagreement resolution such as the one we describe.