Axioms for Retrieval-Augmented Generation
Jan Heinrich Reimer, Maik Fröbe, Benno Stein, Martin Potthast, Matthias Hagen · 2025
Information retrieval axioms are formalized constraints that good retrieval models should fulfill, e.g., to rank documents higher that contain the query terms more often. Over the last decades, more than 25 such axioms have been proposed and used to analyze, to improve, or to explain retrieval models. However, those axioms were meant for document ranking scenarios and thus do not directly fit the new scenario of retrieval-augmented generation systems (RAG). To close this gap, we rethink retrieval axioms for RAG. First, we transfer the underlying ideas of as many of the traditional axioms as possible to the new RAG setting (18 axioms can be transferred), and second, we suggest and formalize 11 new axioms to capture utility aspects of RAG answers. In experiments on the TREC 2024 RAG track data and on the Webis-CrowdRAG-25 corpus, we show that the new axioms more accurately capture automated and human RAG preferences than the transferred traditional axioms. Furthermore, we illustrate practical applications for inspecting preferences of language models and for aiding human preference judgments.