Mining BilingualWord Pairs from Comparable Corpus using Apache Spark Framework
J. P. Sanjanasri, Vijay Krishna Menon, Gadgeon Systems Private Limited, Kochi, Kerala, India, Soman KP, Krzysztof Wołk · 2021
Bilingual dictionaries are essential resources in many areas of natural language processing tasks, but resource-scarce and less popular language pairs rarely have such.Efficient automatic methods for inducting bilingual dictionaries are needed as manual resources and efforts are scarce for low-resourced languages.In this paper, we induce word translations using bilingual embedding.We use the Apache Spark ® framework for parallel computation.Further, to validate the quality of the generated bilingual dictionary, we use it in a phrase-table aided Neural Machine Translation (NMT) system.The system can perform moderately well with a manual bilingual dictionary; we change this into our inducted dictionary.The corresponding translated outputs are compared using the Bilingual Evaluation Understudy (BLEU) and Rank-based Intuitive Bilingual Evaluation Score (RIBES) metrics.