Revolutionizing RAG with Confliction Detection: Transforming Customer Service Chatbots with Large Language Models
Harrison Wu, Wu Haihua · 2025
Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with external knowledge, enabling these models to handle a wide range of domain-specific tasks. One of RAG's real-world applications is in customer service chatbots, where RAG has the potential to increase efficiency while reducing costs. Large businesses that have organized and accurate datasets are able to use RAG efficiently. However, smaller businesses face challenges utilizing RAG as they only have customer service chat history data that may contain incorrect information. To address this limitation, this paper proposes a novel approach that incorporates an adjustable similarity threshold for document retrieval, allowing for the retrieval of a broader set of semantically relevant documents, instead of only the top matches. Furthermore, the method uses LLM capabilities to address answer conflicting and incomplete answers The addition of a voting system after answers are detected to be conflicting is effective in wrong answer removal. As a result, the proposed algorithm significantly improves answer precision, achieving 80.1 %, compared to the base algorithm that achieves a precision of 66.8 %.