Comparison of visual features for audio-visual speech recognition using the AURORA-2J-AV database
Takahiro Togo, Yukitaka Nimura, Takayuki Kitasaka, Kensaku Mori, Yasuhito Suenaga, Chiyomi Miyajima, Kazuya Takeda · The Journal of the Acoustical Society of America · 2006
Various techniques for noise-robust speech recognition have been developed for many years. However, speech recognition in nonstationary noisy environments, such as in car conditions, is still a very challenging research problem. Audio-visual speech recognition is a promising technique for improving noise robustness in such adverse acoustic conditions. Our research group has recently developed an audio-visual speech database ‘‘AURORA-2J-AV’’ following the AURORA2 database protocol, which has been widely used for evaluating the performance of audio-only speech recognition techniques. The AURORA-2J-AV database consists of audio speech signals and two types of image sequences: color and infrared human face images recorded when multi-digit numbers are spoken by about 100 native Japanese speakers. Visual noise as well as acoustic noise is added to the database after recording for simulating various conditions. Audio-visual speech recognition experiments are conducted using the audio speech and infrared images of the AURORA-2J-AV database. Two visual features, correlation coefficients (CCs) between two successive image frames and principal component scores (PCSs) of lip images, are compared in various signal-to-noise ratio (SNR) conditions. Experimental results show that CCs outperform PCSs at high SNRs, while PCSs function better at lower SNRs.