TranSTYLer: Multimodal behavioural style transfer for facial and body gestures generation
Mireille Fares, Catherine Pélachaud, Nicolas Obin · Speech Communication · 2025
This paper addresses the challenge of transferring the behaviour expressivity style of a virtual agent to another one while preserving behaviour shape as they carry communicative meaning. Behaviour expressivity style is viewed here as the qualitative properties of behaviours. We propose TranSTYLer , a multimodal transformer-based model that synthesises the multimodal behaviours of a source speaker with the style of a target speaker. We assume that behaviour expressivity style is encoded across various modalities of communication, including text, speech, body gestures, and facial expressions. The model employs a style-content disentanglement schema to ensure that the transferred style does not interfere with the meaning conveyed by the source’s behaviours. Our approach eliminates the need for style labels and allows the generalisation of styles not seen during the training phase. We train our model on the PATS corpus , which we extended to include dialogue acts and 2D facial landmarks. Objective and subjective evaluations show that our model outperforms state-of-the-art models in style transfer for both seen and unseen styles during training. To tackle the issues of style and content leakage that may arise, we propose a methodology to assess the degree to which behaviour and gestures associated with the target style are successfully transferred while ensuring the preservation of the ones related to the source content.