Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
Mingqi Li, Fei Ding, Dan Zhang, Long Cheng, Hongxin Hu, Feng Luo · 2022
Pre-trained multilingual language models play an important role in cross-lingual natural language understanding tasks.However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their performance.In this paper, we propose Multi-level Multilingual Knowledge Distillation (MMKD), a novel method for improving multilingual language models.Specifically, we employ a teacher-student framework to adopt rich semantic representation knowledge in English BERT.We propose token-, word-, sentence-, and structure-level alignment objectives to encourage multiple levels of consistency between source-target pairs and correlation similarity between teacher and student models.We conduct experiments on crosslingual evaluation benchmarks including XNLI, PAWS-X, and XQuAD.Experimental results show that MMKD outperforms other baseline models of similar size on XNLI and XQuAD and obtains comparable performance on PAWS-X.Especially, MMKD obtains significant performance gains on low-resource languages.