Crowdsourced Assessment of Speech Synthesis
Sabine Buchholz, Javier Latorre, Kayoko Yanagisawa · 2013
This chapter is mainly about crowdsourcing assessment of text-to-speech (TTS) systems, but it also presents some results from attempts to use it for other purposes related to TTS development. Overall assessment of TTS systems is still carried out by asking human subjects to take part in listening tests, which involves listening to synthesized speech samples and answering questions about them. The nature of samples as well as the types of questions/answers vary according to the type of listening test; these are discussed in more detail in the first section. The analysis of crowdsourcing listening tests is built on multiple languages, test types, platforms/labor channels, interface designs, and over a longer time span. The second section reports which approaches worked and which did not. The third section reviews related work on detection and prevention of spamming, while the fourth section develops dedicated metrics for a specific case.