Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks
Bo Peng, Emmanuele Chersoni, Yu-Yin Hsu, Chu‐Ren Huang · 2021
With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures.In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain.By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option.Domain-specific pretraining from scratch, conversely, seems to be less effective.