T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk · 2022
We present a new approach to perform zeroshot cross-modal transfer between speech and text for translation tasks.Multilingual speech and text are encoded in a joint fixed-size representation space.Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities.All our models are trained without the need of cross-modal labeled translation data.Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks.In particular, we outperform the state of the art for zero-shot speech translation on Must-C.We also introduce the first results for zero-shot direct speechto-speech and text-to-speech translation.