MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling
Parishad BehnamGhader, Hossein Zakerinia, Mahdieh Soleymani Baghshah · 2021
Pre-trained models like Bidirectional Encoder Representations from Transformers (BERT), have recently made a big leap forward in Natural Language Processing (NLP) tasks.However, there are still some shortcomings in the Masked Language Modeling (MLM) task performed by these models.In this paper, we first introduce a multi-graph including different types of relations between words.Then, we propose Multi-Graph augmented BERT (MG-BERT) model that is based on BERT.MG-BERT embeds tokens while taking advantage of a static multi-graph containing global word co-occurrences in the text corpus beside global real-world facts about words in knowledge graphs.The proposed model also employs a dynamic sentence graph to capture local context effectively.Experimental results demonstrate that our model can considerably enhance the performance in the MLM task.