CS-BERT: a pretrained model for customer service dialogues
Peiyao Wang, Joyce Fang, Julia Reinspach · 2021
Large-scale pretrained transformer models have demonstrated state-of-the-art (SOTA) performance in a variety of NLP tasks.Nowadays, numerous pretrained models are available in different model flavors and different languages, and can be easily adapted to one's downstream task.However, only a limited number of models are available for dialogue tasks, and in particular, goal-oriented dialogue tasks.In addition, the available pretrained models are trained on general domain language, creating a mismatch between the pretraining language and the downstream domain launguage.In this contribution, we present CS-BERT, a BERT model pretrained on millions of dialogues in the customer service domain.We evaluate CS-BERT on several downstream customer service dialogue tasks, and demonstrate that our indomain pretraining is advantageous compared to other pretrained models in both zero-shot experiments as well as in finetuning experiments, especially in a low-resource data setting.