Audiovisual speech inversion by switching dynamical modeling governed by a Hidden Markov process
Athanasios Katsamanis, G. Ananthakrishnan, George Papandreou, Petros A. Maragos, Olov Engwall · 2008
We propose a unified framework to recover articulation from au-diovisual speech. The nonlinear audiovisual-to-articulatory map-ping is modeled by means of a switching linear dynamical system. Switching is governed by a state sequence determined via a Hid-den Markov Model alignment process. Mel Frequency Cepstral Coefficients are extracted from audio while visual analysis is per-formed using Active Appearance Models. The articulatory state is represented by the coordinates of points on important articula-tors, e.g., tongue and lips. To evaluate our inversion approach, in-stead of just using the conventional correlation coefficients and root mean squared errors, we introduce a novel evaluation scheme that is more specific to the inversion problem. Prediction errors in the posi-tions of the articulators are weighted differently depending on their relevant importance in the production of the corresponding sound. The applied weights are determined by an articulatory classification analysis using Support Vector Machines with a radial basis function kernel. Experiments are conducted in the audiovisual-articulatory MOCHA database. 1.