NICT’s Cascaded and End-To-End Speech Translation Systems using Whisper and IndicTrans2 for the Indic Task
Raj Dabre, Haiyue Song · 2024
This paper presents the NICT's submission for the IWSLT 2024 Indic track, focusing on three speech-to-text (ST) translation directions: English to Hindi, Bengali, and Tamil.We aim to enhance translation quality in this low-resource scenario by integrating state-ofthe-art pre-trained automated speech recognition (ASR) and text-to-text machine translation (MT) models.Our cascade system incorporates a Whisper model fine-tuned for ASR and an IndicTrans2 model fine-tuned for MT.Additionally, we propose an end-to-end system that combines a Whisper model for speech-totext conversion with knowledge distilled from an IndicTrans2 MT model.We first fine-tune the IndicTrans2 model to generate pseudo data in Indic languages.This pseudo data, along with the original English speech data, is then used to fine-tune the Whisper model.Experimental results show that the cascaded system achieved a BLEU score of 51.0, outperforming the end-to-end model, which scored 19.1 BLEU.Moreover, the analysis indicates that applying knowledge distillation from the In-dicTrans2 model to the end-to-end ST model improves the translation quality by about 0.7 BLEU.