Effectively Crowdsourcing Radiology Report Annotations
Anne Cocos, Aaron J. Masino, Ting Qian, Ellie Pavlick, Chris Callison-Burch · 2015
Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale.We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge.Comparing a sentence classification model trained with expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset.We can improve the accuracy of the crowd-fueled model without collecting further labels by filtering out worker labels applied with low confidence.