Advancing Objective Evaluation of Speech-Driven Gesture Generation for Embodied Conversational Agents
Karlo Crnek, Grega Močnik, Matej Rojc · International Journal of Human-Computer Interaction · 2025
Developing effective speech-driven gesture generation models for embodied conversational agents is crucial for creating engaging human-agent interactions. However, the current reliance on costly and time-consuming subjective evaluation studies poses challenges in confident model comparison and efficient development. To address this, we propose novel feature extractors for a similarity-based objective evaluation framework, designed to enhance correlation with subjective studies. To validate new approaches, we conducted an extensive empirical analysis of 56 combinations of feature extractors and distance measures using data from the GENEA 2022 and 2023 challenges. Our results show that a CNN-based feature extractor trained on motion-audio sync classification, combined with Fréchet distance, achieved the highest correlation (0.809) with the GENEA subjective studies. This highlights the advantage of neural network-based feature extractors in the objective evaluation pipeline, emphasizing the critical role of the training task, as it outperformed feature extractors trained on alternative tasks.