Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study
Aniruddha Roy, Pretam Ray, Ayush Maheshwari, Sudeshna Sarkar, Pawan Goyal · 2024
Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages.Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated impressive performance in various lowresource NMT tasks.However, their pretraining has been confined to 50 languages, leaving out support for numerous low-resource languages, particularly those spoken in the Indian subcontinent.Expanding mBART-50's language support requires complex pretraining, risking performance decline due to catastrophic forgetting.Considering these expanding challenges, this paper explores a framework that leverages the benefits of a pre-trained language model along with knowledge distillation in a seq2seq architecture to facilitate translation for low-resource languages, including those not covered by mBART-50.The proposed framework employs a multilingual encoderbased seq2seq model as the foundational architecture and subsequently uses complementary knowledge distillation techniques to mitigate the impact of imbalanced training.Our framework is evaluated on three low-resource Indic languages in four Indic-to-Indic directions, yielding significant BLEU-4 and chrF improvements over baselines 1 .Further, we conduct human evaluation to confirm effectiveness of our approach.