CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer
Sri Karlapati, Penny Karanasou, Mateusz Łajszczak, Syed Ammar Abbas, Alexis Moinet, Peter Makarov, Ray Li, Arent van Korlaar, Simon Slangen, Thomas Drugman · Interspeech 2022 · 2022
In this paper, we present CopyCat2 (CC2), a novel model capable of: a) synthesizing speech with different speaker identities, b) generating speech with expressive and contextually appropriate prosody, and c) transferring prosody at fine-grained level between any pair of seen speakers.We do this by activating distinct parts of the network for different tasks.We train our model using a novel approach to two-stage training.In Stage I, the model learns speaker-independent word-level prosody representations from speech which it uses for many-to-many finegrained prosody transfer.In Stage II, we learn to predict these prosody representations using the contextual information available in text, thereby, enabling multi-speaker TTS with contextually appropriate prosody.We compare CC2 to two strong baselines, one in TTS with contextually appropriate prosody, and one in fine-grained prosody transfer.CC2 reduces the gap in naturalness between our baseline and copy-synthesised speech by 22.79%.In fine-grained prosody transfer evaluations, it obtains a relative improvement of 33.15% in target speaker similarity.