EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Chen Lin, Timothy M. Miller, Dmitriy Dligach, Steven J. Bethard, Guergana Savova · 2021
Transformer-based neural language models have led to breakthroughs for a variety of natural language processing (NLP) tasks.However, most models are pretrained on general domain data.We propose a methodology to produce a model focused on the clinical domain: continued pretraining of a model with a broad representation of biomedical terminology (PubMed-BERT) on a clinical corpus along with a novel entity-centric masking strategy to infuse domain knowledge in the learning process.We show that such a model achieves superior results on clinical extraction tasks by comparing our entity-centric masking strategy with classic random masking on three clinical NLP tasks: cross-domain negation detection (Wu et al., 2014), document time relation (Doc-TimeRel) classification (Lin et al., 2020b), and temporal relation extraction (Wright-Bettner et al., 2020).We also evaluate our models on the PubMedQA (Jin et al., 2019) dataset to measure the models' performance on a nonentity-centric task in the biomedical domain.The language addressed in this work is English.