Reducing named entity hallucination risk to ensure faithful summary generation
Eunice Akani, Benoît Favre, Frédéric Béchet, Romain Gemignani · 2023
The faithfulness of abstractive text summarization at the named entities level is the focus of this study.We propose to add a new criterion to the summary selection method based on the "risk" of generating entities that do not belong to the source document.This method is based on the assumption that Out-Of-Document entities are more likely to be hallucinations.This assumption was verified by a manual annotation of the entities occurring in a set of generated summaries on the CNN/DM corpus.This study showed that only 29% of the entities outside the source document were inferrable by the annotators, leading to 71% of hallucinations among OOD entities.We test our selection method on the CNN/DM corpus and show that it significantly reduces the hallucination risk on named entities while maintaining competitive results with respect to automatic evaluation metrics like ROUGE.