Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy
Liyan Xu, Zhenlin Su, Mo Yu, Jin Xu, Jinho D. Choi, Jie Na Zhou, Liu Fei · 2024
Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models.While a major direction to enhance inconsistency detection is to derive stronger Natural Language Inference (NLI) models, we propose an orthogonal aspect that underscores the importance of incorporating task-specific taxonomy into the inference.To this end, we consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.Extensive experiments on ten datasets of five distinct domains suggest that, zero-shot LLM inference could benefit from the explicit solution space depicted by the error type taxonomy, and achieves stateof-the-art performance overall, surpassing specialized non-LLM baselines, as well as recent LLM baselines.We further distill models that fuse the taxonomy into parameters through our designed prompt completions and supervised training strategies, efficiently substituting stateof-the-art zero-shot inference with much larger LLMs.Our data and code are publicly released at https://github.com/lxucs/factax.