Using Amazon Mechanical Turk for Transcription of Non-Native Speech
Keelan Evanini, Derrick Higgins, Klaus Zechner · 2010
This study investigates the use of Amazon Mechanical Turk for the transcription of non-native speech. Multiple transcriptions were obtained from several distinct MTurk workers and were combined to produce merged tran-scriptions that had higher levels of agreement with a gold standard transcription than the in-dividual transcriptions. Three different meth-ods for merging transcriptions were compared across two types of responses (spontaneous and read-aloud). The results show that the merged MTurk transcriptions are as accurate as an individual expert transcriber for the read-aloud responses, and are only slightly less ac-curate for the spontaneous responses. 1