GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation
Igor Shalyminov, Alessandro Sordoni, Adam Atkinson, Hannes Schulz · IEEE/ACM Transactions on Audio Speech and Language Processing · 2021
Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in real-world scenarios, such resources are rarely available for new domains, and the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we presentgrtr, a hybrid generative-retrieval model based on the large-scale general-purpose language model GPT[2] fine-tuned to the multi-domainmetalwoz dataset. In addition to robust and diverse response generation provided by the GPT[2], our model is able to estimate generation confidence, and is equipped with retrieval logic as a fallback for the cases when the estimate is low.grtr is the winning entry at the fast domain adaptation task of DSTC-8 in human evaluation ($>$4% improvement over the 2nd place system). It also attains superior performance to a series of baselines on automated metrics onmetalwoz andmultiwoz, a multi-domain dataset of goal-oriented dialogues. In this paper, we also conduct a study ofgrtr's performance in the setup of limited adaptation data, evaluating the model's overall response prediction performance onmetalwoz and goal-oriented performance onmultiwoz.