Crowdsourcing and language studies: the new generation of linguistic data
Robert Munro, Steven J. Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher E. Potts, Tyler Schnoebelen, Harry J. Tily · Max Planck Institute for Plasma Physics · 2010
We present a compendium of recent and current projects that utilize crowdsourcing technologies for language studies, finding that the quality is comparable to controlled laboratory experiments, and in some cases superior. While crowdsourcing has primarily been used for annotation in recent language studies, the results here demonstrate that far richer data may be generated in a range of linguistic disciplines from semantics to psycholinguistics. For these, we report a number of successful methods for evaluating data quality in the absence of a 'correct' response for any given data point.