Preliminary Experiments with Amazon's Mechanical Turk for Annotating Medical Named Entities
Meliha Yetisgen-Yildiz, Imre Solti, Fei Xia, Scott Halgrim · 2010
Amazon’s Mechanical Turk (MTurk) service is becoming increasingly popular in Natural Language Processing (NLP) research. In this paper, we report our findings in using MTurk to annotate medical text extracted from clinical trial descriptions with three entity types: medical condition, medication, and laboratory test. We compared MTurk annotations with a gold standard manually created by a domain expert. Based on the good performance results, we conclude that MTurk is a very promising tool for annotating large-scale corpora for biomedical NLP tasks. 1