Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk
Joel Tetreault, Elena Vladimirovna Filatova, Martin S. Chodorow · 2010
hunter.cuny.edu In this paper we present results from two pi-lot studies which show that using the Amazon Mechanical Turk for preposition error anno-tation is as effective as using trained raters, but at a fraction of the time and cost. Based on these results, we propose a new evaluation method which makes it feasible to compare two error detection systems tested on different learner data sets. 1