Multi-Agent Based Casual Triple Extraction For Factuality Evaluation Using Large Language Models
Jian Zhang, Zemin Xu, Lifang Liu, Zhanfeng Shen, Yue Cui, Yongdong Zhang · 2024
Ensuring the factual accuracy of content generated by large language models (LLMs) is of paramount importance for applications in domains such as finance, healthcare, and education, where reliability and trustworthiness are critical. In this paper, we propose a multi-agent pipeline framework (MAPF) that integrates content generation, fact extraction, and factuality verification into a structured and robust process. The pipeline leverages five specialized agents: Question Parse Agent, Search Agent, Answer Generation Agent, Fact Description Extraction Agent, and Factuality Judge Agent, each responsible for a specific task to ensure both coherence and factual consistency. By extracting and evaluating factual content through a novel structured representation based on causal triples, we introduce an approach that significantly improves the precision of factual judgments. Experimental results on a custom dataset and the public FiQA dataset demonstrate the effectiveness of the proposed pipeline in improving factual consistency by leveraging structured causal fact representations, especially in complex, knowledge-intensive domains. Our findings suggest that structured fact extraction and agent-based task specialization offer a promising pathway for enhancing LLM factuality across various applications.