Using tone similarity judgments in tests of intertranscriber reliability
Julia Tevis McGory, Rebecca Herman, Ann K. Syrdal · The Journal of the Acoustical Society of America · 1999
An important, albeit often neglected, aspect of intonational labeling is testing the consistency between labelers. When reliability among different labelers has been tested, two aspects in which transcribers differ from each other is in placement and choice of tone labels. Only identical labels count as ‘‘agreement’’ between transcriptions. This can be problematic given that some tone categories are perceived to be similar, making the choice between labels at times difficult. This work suggests a refinement of existing tests of intertranscriber reliability. Previous intertranscriber reliability tests did not take into account each labeler’s perceived similarity between pitch accent categories. We examined a set of ToBI transcriptions of a single speaker corpus labeled independently by four experienced intonation labelers. The proposed method incorporates each transcriber’s tone similarity judgments, which were elicited for the purposes of this experiment. The labelers judged the similarity of pitch accent pairs using a rating scale of 1–7. Multidimensional scaling (MDS) of similarity judgements resulted in quantifiable measures of degree of similarity among tones. Euclidean distances between pairs of tones were calculated from the MDS results. We used these values as the weights in calculating mean weighted agreements. [This work is supported by AT&T Labs.–Research.]