Domain Adaptation with BERT-based Domain Classification and Data Selection
Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh M. Nallapati, Bing Xiang · 2019
The performance of deep neural models can deteriorate substantially when there is a domain shift between training and test data.For example, the pre-trained BERT model can be easily fine-tuned with just one additional output layer to create a state-of-the-art model for a wide range of tasks.However, the fine-tuned BERT model suffers considerably at zero-shot when applied to a different domain.In this paper, we present a novel two-step domain adaptation framework based on curriculum learning and domain-discriminative data selection.The domain adaptation is conducted in a mostly unsupervised manner using a small target domain validation set for hyper-parameter tuning.We tested the framework on four large public datasets with different domain similarities and task types.Our framework outperforms a popular discrepancy-based domain adaptation method on most transfer tasks while consuming only a fraction of the training budget.