Model-based lip synchronization with automatically translated synthetic voice toward a multi-modal translation system
Shin Ogata, Kazumasa Murai, Satoshi Nakamura, Shigeo Morishima · 2001
In this paper, we introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database.