English-Assamese Multimodal Neural Machine Translation using Transliteration-based Phrase Augmentation Approach
Sahinur Rahman Laskar, Bishwaraj Paul, Partha Pakray, Sivaji Bandyopadhyay · Procedia Computer Science · 2023
Neural machine translation (NMT) is a popular machine translation method due to its contextual analyzing ability and end-to-end process flexibility. However, NMT suffers poor translation quality in low-resource contexts, particularly for diverse language pairs. To overcome this issue, multimodal concept has been introduced in NMT, wherein leverage information from different modalities like image or speech in addition to text to enhance automatic translation quality. In this paper, we have investigated multimodal NMT for a low-resource language diverse pair, English-Assamese, by addressing data scarcity and word-order divergence issues. To tackle such issues, a transliteration-based phrase augmentation approach is proposed, that leverages the sub-word level tokens sharing among source-target sequences in the training process via transliteration and provides more word alignment information by the addition of phrase pairs. Also, the relevant image features corresponding to the phrase pairs are augmented by considering a filtering step. With the proposed approach, state-of-the-art multimodal NMT results are attained for both directions of English-Assamese pair translation.