Construction of Domain-Specific DistilBERT Model by Using Fine-Tuning
Jing Bai, Rui Cao, Wen Ma, Hiroyuki Shinnou · 2020
In this paper, we point out the problem that BERT is domain dependent, and propose to construct the domain specific pre-training DistilBERT model by using fine-tuning. In particular, parameters of a DistilBERT model are initialized using a trained BERT model, and then these parameters are tuned from the specific domain corpus. Consequently, we can construct efficiently the domain-specific DistilBERT model. In the experiment, we experiment the test set for each domain; the estimation of a masked word in a sentence. Through this experiment, we show the superiority of our proposed model (domain-specific DistilBERT model) by comparing it with the general BERT model.