MDG:Multilingual Co-speech Gesture Generation with Low-level Audio Representation and Diffusion Models
Jie Yang, Feilong Bao · 2024
Co-speech gesture generation is crucial for enhancing human-computer interaction. Most existing research focuses on English, overlooking multilingual speech and the ability to generalize across languages. Additionally, these studies often rely on pre-trained audio feature extraction models, insufficient for non-English languages. To address these issues, we introduce MDG, a novel diffusion model-based method for speech-driven gesture generation using a multilingual dataset. Our approach incorporates Language ID to distinguish between languages and utilizes low-level audio features to improve multilingual generalization. Extensive experiments demonstrate that MDG significantly outperforms baseline models in multilingual co-speech gesture generation. For more details, please visit our demo page1.