From Text to Multi-Modal: Advancing Low-Resource-Language Translation through Synthetic Data Generation and Cross-Modal Alignments
Bushi Xiao, Qian Shen, Daisy Zhe Wang · 2025
In this study, we propose a novel paradigm for multi-modal low resource language dataset generation without relying on existing parallel multi-modal datasets.Leveraging advances in image-generation models, we introduce a systematic pipeline that transforms text-only parallel corpora into rich multimodal translation datasets.We then validate the generated content through human evaluation.We design and implement a new MMT model framework suitable for our new generated dataset.The model contains a verification mechanism with a large language model to ensure consistency between visual content and textual translations.Experimental results across four African lowresource languages with less than 10k training corpus demonstrate significant improvements over NLLB baselines, with average gains of up to 9.8% in BLEU score and 4.3% in METEOR score.Our method shows particular effectiveness in correctly translating concrete objects and contextual elements, suggesting its potential for improving low-resource machine translation through visual grounding.