WeLT: Improving Biomedical Fine-tuned Pre-trained Language Models with Cost-sensitive Learning
Ghadeer Mobasher, Wolfgang Müller, Olga Krebs, Michael Gertz · 2023
Fine-tuning biomedical pre-trained language models (BioPLMs) such as BioBERT has become a common practice dominating leaderboards across various natural language processing tasks.Despite their success and wide adoption, prevailing fine-tuning approaches for named entity recognition (NER) naively train BioPLMs on targeted datasets without considering class distributions.This is problematic especially when dealing with imbalanced biomedical gold-standard datasets for NER in which most biomedical entities are underrepresented.In this paper, we address the class imbalance problem and propose WeLT, a costsensitive fine-tuning approach based on new rescaled class weights for the task of biomedical NER.We evaluate WeLT's fine-tuning performance on mixed-domain and domain-specific BioPLMs using eight biomedical gold-standard datasets.We compare our approach against vanilla fine-tuning and three other existing reweighting schemes.Our results show the positive impact of handling the class imbalance problem.WeLT outperforms all the vanilla fine-tuned models.Furthermore, our method demonstrates advantages over other existing weighting schemes in most experiments.