Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data
Natalie Parde, Rodney D. Nielsen · 2017
Crowdsourcing offers a convenient means of obtaining labeled data quickly and inexpensively.However, crowdsourced labels are often noisier than expert-annotated data, making it difficult to aggregate them meaningfully.We present an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications.The predicted labels achieve a correlation of 0.594 with expert labels on our data, outperforming the best alternative aggregation method by 11.9%.Our approach also outperforms the alternatives on third-party datasets.