Multilingual Fine-Grained News Headline Hallucination Detection
Jiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin, Jay Pavagadhi, Simon Baumgartner, Michael Bendersky · 2024
The popularity of automated news headline generation has surged with advancements in pre-trained language models.However, these models often suffer from the "hallucination" problem, where the generated headline is not fully supported by its source article.Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types.In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand article, headline pairs in 5 languages, each annotated with detailed hallucination types by experts.We conduct extensive experiments on this dataset under two settings.First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities.Second, we test various large language models' in-context learning abilities and propose two novel techniques, languagedependent demonstration selection and coarseto-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric.We release this dataset to foster further research in multilingual, finegrained headline hallucination detection.