A non-deteriorating approach to improve Natural Language Understanding in Conversational Agents
Miguel Arevalillo‐Herráez, Romina Soledad Albornoz-De Luise, Yuyan Wu · Neurocomputing · 2025
The performance of Conversational Agents (CAs) relies heavily on intent classification and entity extraction. However, the effectiveness of these tasks is often hindered by the absence of an explicit model that accounts for their interdependence. To address this limitation, we introduce a novel approach that leverages the inter-dependency between intents and entities to improve the Exact Match Accuracy (EMA) of CAs. The approach evaluates the consistency of the output generated by the Natural Language Understanding (NLU) component based on a specification that outlines all valid combinations of intents and entities. If an inconsistent output is detected, the ranking of predicted intents is leveraged to determine the most probable intent that aligns with the identified entities. The technique guarantees the preservation or improvement of EMA and can be applied as a post-processing step in combination with any existing NLU method that returns a ranking of intents. The proposed approach was evaluated through an ablation study conducted using three different NLU models (DIET, DCA-Net and Bi-model) on four distinct conversational datasets: ATIS, SNIPS, NLU-Benchmark, and AWPS. The results demonstrated that our method led to a consistent improvement in the EMA across all NLU methods and datasets. The magnitude of the improvement varies depending on the method, dataset, and training data size, ranging from below 1% when using DCA-Net or Bi-model on the SNIPS dataset with full training data, to over 9% when using Bi-model or DIET on AWPS with 10% of the available training data. • A novel approach to increase the exact match accuracy in NLU systems is presented. • The method is non-deteriorating and seamlessly integrates with existing NLU systems. • The technique leverages the inherent inter-dependency between intents and entities. • It amends incompatible outputs by using a human-provided constraint specification. • The system’s performance consistently improved in 4 commonly used NLU datasets.