Audio visual interaction in multimedia
Tsuhan Chen, Ram C. Rao · IEEE Circuits and Devices Magazine · 1995
Audio-visual interaction is a very important issue in personal communication applications. The research and development of multimedia communication systems should account for this interaction. In this paper, we address a number of areas related to audio-visual interaction, such as automatic lipreading, speech-driven talking heads, and lip synchronization. In particular, we will discuss a new trend in video coding research: joint audio-video coding. Given that mouth movements are very difficult to code because of its rapid, complex, and non-rigid motion (so conventional block-based motion-compensation methods fail), we will explain how having extra help from the acoustic signal can enable us to code the mouth movements more efficiently.