A Multimodal Listener Behaviour Driven by Audio Input
Étienne de Sevin, Elisabetta Bevacqua, Sathish Pammi, Catherine Pélachaud, Markus Schröder, Björn Wolfgang Schuller · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · 2010
Our aim is to build a platform allowing a user to chat with virtual agent. The agent displays audio-visual backchannels as a response to the user’s verbal and nonverbal behaviours. Our system takes as inputs the audio-visual signals of the user and outputs synchronously the audio-visual behaviours of the agent. In this paper, we describe the SEMAINE architecture and the data flow that goes from inputs (audio and video) to outputs (voice synthesizer and virtual characters), going through analysers and interpreters. We focus, more particularly, on the multimodal behaviour of the listener model driven by audio input.