Entity Enhanced BERT Pre-training for Chinese NER
Jia Chen, Yuefeng Shi, Qinrong Yang, Yue Zhang · 2020
Character-level BERT pre-trained in Chinese suffers a limitation of lacking lexicon information, which shows effectiveness for Chinese NER.To integrate the lexicon into pre-trained LMs for Chinese NER, we investigate a semisupervised entity enhanced BERT pre-training method.In particular, we first extract an entity lexicon from the relevant raw text using a newword discovery method.We then integrate the entity information into BERT using Char-Entity-Transformer, which augments the selfattention using a combination of character and entity representations.In addition, an entity classification task helps inject the entity information into model parameters in pre-training.The pre-trained models are used for NER finetuning.Experiments on a news dataset and two datasets annotated by ourselves for NER in long-text show that our method is highly effective and achieves the best results.