On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov · 2020
Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages.However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference.In this paper, we present the first systematic study of negative interference.We show that, contrary to previous belief, negative interference also impacts low-resource languages.While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference.Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding languagespecific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages.Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations. 1 Model NER (F1) POS (F1) ar fr ru hi sw te avg