An Active Learning Pipeline for NLU Error Detection in Conversational Agents
Damián Pascual, Aritz Bercher, Akansha Bhardwaj, Mingbo Cui, Dominic Kohler, Liam van der Poel, Paolo Rosso · 2023
High-quality labeled data is paramount to the performance of modern machine learning models.However, annotating data is a timeconsuming and costly process that requires human experts to examine large collections of raw data.For conversational agents in production settings with access to large amounts of user-agent conversations, the challenge is to decide what data should be annotated first.We consider the Natural Language Understanding (NLU) component of a conversational agent deployed in a real-world setup with limited resources.We present an active learning pipeline for offline detection of classification errors that leverages two strong classifiers.Then, we perform topic modeling on the potentially mis-classified samples to ease data analysis and to reveal error patterns.In our experiments, we show on a realworld dataset that by using our method to prioritize data annotation we reach 100% of the performance annotating only 36% of the data.Finally, we present an analysis of some of the error patterns revealed and argue that our pipeline is a valuable tool to detect critical errors and reduce the workload of annotators.