Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Dengyong Zhou, Qiang Liu, John C. Platt, Christopher Meek · 2014
We propose a method to aggregate noisy ordi-nal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and rating products. Our method is motivated by the observation that workers usually have diffi-culty distinguishing between two adjacent ordi-nal classes whereas distinguishing between two classes which are far away from each other is much easier. We formulate our method as min-imax conditional entropy subject to constraints which encode this observation. Empirical eval-uations on real datasets demonstrate significant improvements over existing methods. 1.