Who’s Who: Large Language Models Meet Knowledge Conflicts in Practice
Quang Hiếu Phạm, Hoang Ngo, Anh Tuan Luu, Dat Quoc Nguyen · 2024
Retrieval-augmented generation (RAG) methods are viable solutions for addressing the static memory limits of pre-trained language models.Nevertheless, encountering conflicting sources of information within the retrieval context is an inevitable practical challenge.In such situations, the language models are recommended to transparently inform users about the conflicts rather than autonomously deciding what to present based on their inherent biases.To analyze how current large language models (LLMs) align with our recommendation, we introduce WhoQA, a public benchmark dataset to examine model's behavior in knowledge conflict situations.We induce conflicts by asking about a common property among entities having the same name, resulting in questions with up to 8 distinctive answers.WhoQA evaluation set includes 5K questions across 13 Wikidata property types and 150K Wikipedia entities.Our experiments show that despite the simplicity of WhoQA questions, knowledge conflicts significantly degrades LLMs' performance in RAG settings. * The first two authors contributed equally to this work.What is George Washington's occupation?George Washington (February 22, 1732 -December 14, 1799) was an American Founding Farther, military officer, and politician who served as the first president of the United States from 1789 to 1797.George Washington (May 20, 1871 -March 29, 1946) was a Belgium-born American inventor.George Washington (born October 18, 1907) was an American jazz trombonist.George Washington's occupation was primarily as a military officer, statesman, and politician.He served as a