Audio and visual modality combination in speech processing applications
Gerasimos Potamianos, Etienne Marcheret, Youssef Mroueh, Vaibhava Goel, Alexandros Koumbaroulis, Argyrios Vartholomaios, Spyridon Thermos · ACM eBooks · 2017
Chances are that most of us have experienced difficulty in listening to our interlocutor during face-to-face conversation while in highly noisy environments, such as next to heavy traffic or over the background of high-intensity speech babble or loud music. In such occasions, we may have found ourselves looking at the speaker's lower face, while our interlocutor articulates speech, in order to help us enhance speech intelligibility. In fact, what we resort to in such circumstances is known as lipreading or speechreading, namely the recognition of the so-called "visual speech modality" and its combination (fusion) with the available noisy audio data.