Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure
Yi Chai, Yaobo Liang, Nan Duan · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2022
Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability.Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervision or aligned data.Despite the encouraging results, we still lack a clear understanding of why cross-lingual ability could emerge from multilingual MLM.In our work, we argue that crosslanguage ability comes from the commonality between languages.Specifically, we study three language properties: constituent order, composition and word co-occurrence.First, we create an artificial language by modifying property in source language.Then we study the contribution of modified property through the change of cross-language transfer results on target language.We conduct experiments on six languages and two cross-lingual NLP tasks (textual entailment, sentence retrieval).Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.