Correcting Factuality Hallucination in Complaint Large Language Model via Entity-Augmented
Jiaju Kang, Weichao Pan, Tian Zhang, Ziming Wang, Shuqin Yang, Zhiqin Wang, Jian Wang, Xiaofei Niu · 2024
Complaint Large Language Model (Complaint-LLM) is designed as a "customer service" tool to address the scenario of handling a massive volume of public complaints, effectively leveraging the "common sense" possessed by Large Language Models (LLMs) to solve issues. Unfortunately, pre-trained LLMs often exhibit significant Factual Hallucination and Causal Errors in knowledge domains with sparse experience distribution, greatly affecting the accuracy of user interactions with LLMs. We propose an architecture that utilizes external data to support pre-trained models, aiming to avoid the expensive cost of retraining LLMs. The core concept involves leveraging prompts to inject strongly correlated additional information into LLMs and adjusting the initialized alternative outputs along the inference pathway of the LLM. To achieve this, we construct a rich knowledge graph as a knowledge base for algorithm retrieval and learning. Each input text is decomposed into subgraphs corresponding to nodes on the knowledge graph, and a graph neural network classifier is trained to obtain classification results and additional knowledge. Numerous experiments demonstrate that the Complaint-LLMs shows a significant improvement in the question-answering evaluation of various subclass scenarios in the complaint domain. Moreover, the graph neural network trained with complaint text data exhibits good transferability in classification tests for open scenarios.