Quality Assessment for Crowdsourced Object Annotations
Sirion Vittayakorn, James H. Hays · 2011
As computer vision datasets grow larger, the community is increasingly relying on crowdsourced annotations to train and test their algorithms. Since the capability of online annotators is considered varied and unpredictable, many strategies have been proposed to “clean ” crowdsourced annotations. However, these strategies typically require more annotations, rather than using the annotation or image content itself. In this paper we propose and evaluate several strategies for automatically estimating the quality of an object annotation. Finally, we show that we can significantly outperform simple baselines by combining multiple image-based annotation assessment strategies. 1