Conversation Clustering Adaptation for Intent Recognition
Michał Lew, Aleksander Obuchowski, Emilia Kacprzak, Agnieszka Pluwak · 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) · 2021
With the increasing presence of NLU tools such as automatic dialogue systems, achieving high accuracy of intent recognition in chatbots becomes an especially important problem to tackle. This issue cannot be solved without sufficient training data, but the scarcity of labelled training data often poses a major challenge to the development of real-life chatbots. Therefore, methods utilizing unlabelled data resources have been recently gaining interest. One of most notable approaches is the use of pre-trained encoders based on language models. Trained for general purposes, they benefit from further domain adjustments. In our work we offer an approach that can increase the model’s accuracy for text classification, which can serve as an alternative for standard methods of domain adaptation. Our approach consists of a combination of methods: a clustering approach, similar to intent induction; an encoder domain adaptation on a cluster classification task, similar to intent recognition using unlabelled data; and model fine-tuning on labelled datasets. In this approach unlabelled data becomes complementary to labelled data, reducing the time needed for corpus building. We evaluate our approach on: 1) the public WebApp dataset and 2) a demanding real-life banking domain dataset, achieving 0.97 and 0.93 accuracy respectively. This approach, called Conversation Clustering Adaptation (CCA), when applied to an encoder, increases the accuracy of intent recognition up by to 12.4pp and exceeds current state-of-the-art methods while benefiting from the use of additional training data. We share our code at https://github.com/michal-lew/cca.