Inter-Annotator Agreement on Spontaneous Czech Language - Limits of Automatic Speech Recognition Accuracy.
Tomáš Valenta, Luboš Šmídl, Ján Švec, Daniel Soutner · Text, Speech and Dialogue · 2014
The goal of this article is to show that for some tasks in automatic speech recognition (ASR), especially for recognition of spontaneous telephony speech, the reference annotation differs substantially among human annotators and thus sets the upper bound of the ASR accuracy. In this paper, we focus on the evaluation of the inter-annotator agreement (IAA) and ASR accuracy in the context of imperfect IAA. We evaluated it using a part of our Czech Switchboard- like spontaneous speech corpus called Toll-free calls. This data set was annotated by three different annotators rendering three parallel transcriptions. The results give us additional insights for understanding the ASR accuracy.