Meaningless yet meaningful: Morphology grounded subword-level NMT
Tamali Banerjee, Pushpak Bhattacharyya · 2018
We explore the use of two independent subsystems, namely Byte Pair Encoding (BPE) and Morfessor as basic units for subword-level neural machine translation (NMT).We have shown that for linguistically distant language-pairs Morfessor-based segmentation algorithm produces significantly better quality translation than BPE.However, for close language-pairs BPE-based subword-NMT may translate better than Morfessor-based subword-NMT.We have proposed a combined approach of these two segmentation algorithms Morfessor-BPE (M-BPE) which outperforms these two baseline systems in terms of BLEU score.Our results are supported by experiments on three language-pairs: English-Hindi, Bengali-Hindi and English-Bengali.