Annotating Large Email Datasets for Named Entity Recognition with Mechanical Turk
Nolan Lawson, Kevin Eustice, Mike Perkowitz, Meliha Yetisgen-Yildiz · 2010
Amazon's Mechanical Turk service has been successfully applied to many natural language processing tasks. However, the task of named entity recognition presents unique challenges. In a large annotation task involving over 20,000 emails, we demonstrate that a competitive bonus system and interannotator agreement can be used to improve the quality of named entity annotations from Mechanical Turk. We also build several statistical named entity recognition models trained with these annotations, which compare favorably to similar models trained on expert annotations. 1