Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, Yulan He · 2021
Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature.Although an increasing number of biomedical QA datasets has been recently made available, those resources are still rather limited and expensive to produce.Transfer learning via pre-trained language models (LMs) has been shown as a promising approach to leverage existing general-purpose knowledge.However, finetuning these large models can be costly and time consuming, often yielding limited benefits when adapting to specific themes of specialised domains, such as the COVID-19 literature.To bootstrap further their domain adaptation, we propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM).We encourage masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning.The resulting strategy is a downstream process applicable to a wide variety of masked LMs, not requiring additional memory or components in the neural architectures.Experimental results show performance on par with state-of-the-art models on several biomedical QA datasets.