Effect of degradations' distribution in a corpus test on auditory ratings
Nicolas Côté, Virginie Durin · The Journal of the Acoustical Society of America · 2008
Speech quality of telecommunications systems is usually evaluated thanks to auditory tests, which must be carried out in accordance with ITU-T Recommendations. In these tests, subjects are asked to assess the quality of speech sample by giving a score on a five-level scale. The averaging of subjects' scores yields the Mean Opinion Score (MOS) which represents the speech quality for a given condition. However, MOS values can be strongly influenced by many factors. In this paper, we focus on a specific bias: the distribution of the impairments in the stimuli set. In effect the rating of a narrow-band stimulus (300-3400 Hz) may have a lower quality score in a mixed-band set (mixed of narrow-band and wideband (50-7000 Hz) conditions) than in a purely narrow band set. This effect is detailed through results of auditory tests recently carried out. Consequently, the validity of MOS value is theoretically limited within a test which prevents MOS comparison between different tests. Finally, two solutions are proposed to avoid this effect: first, an improvement of auditory tests methodology and then, a new approach to assess speech quality, based on the subject's behaviour.