Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
Takashi Wada, Tomoharu Iwata, Yūji Matsumoto, Timothy J. Baldwin, Jey Han Lau · 2021
We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus (e.g. a few hundred sentence pairs).Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.Through sharing model parameters among different languages, our model jointly trains the word embeddings in a common cross-lingual space.We also propose to combine word and subword embeddings to make use of orthographic similarities across different languages.We base our experiments on real-world data from endangered languages, namely Yongning Na, Shipibo-Konibo, and Griko.Our experiments on bilingual lexicon induction and word alignment tasks show that our model outperforms existing methods by a large margin for most language pairs.These results demonstrate that, contrary to common belief, an encoder-decoder translation model is beneficial for learning crosslingual representations even in extremely lowresource conditions.Furthermore, our model also works well on high-resource conditions, achieving state-of-the-art performance on a German-English word-alignment task. 1