Audio quality evaluation by experienced and inexperienced listeners
Nadja Schinkel–Bielefeld, Netaya Lotze, Frederik Nagel · The Journal of the Acoustical Society of America · 2013
Basic audio quality of coded audio material is commonly evaluated using ITU-R BS-1534 Multi Stimulus with Hidden Reference and Anchors (MUSHRA) listening test. MUSHRA guidelines call for experienced listeners. However, the majority of consumers using the final product are no expert-listeners. Also the degree of expertise in a listening test may vary among listeners in the same laboratory. It would be useful to know how the audio quality evaluation differs between trained and untrained listeners and how training and actual tests should be designed in order to be as reliable as possible. To investigate the rating differences between experts and non-experts, we performed MUSHRA listening tests with 13 experienced and 11 inexperienced listeners using 5 speech and audio codecs delivering a wide range of basic audio quality. Except for the hidden reference, absolute ratings of non-experts were consistently at least 10% higher than those of experts. However, they could be mapped to each other by a z-transform. For lower quality values, confidence intervals were significantly larger for non-experts than for experts. Experienced listeners set more than twice as many loops as non-experts, compared more often between codecs and listened to high quality codecs for a longer duration than non-experts.