Normalized Set-Level Ideal DCG: A More Reliable Early-Stage Retrieval Metric
Xiao Fu, Navdeep Singh Bedi, Fábio Crestani, Aldo Lipani · 2025
Multi-stage information retrieval systems often employ a retrieve-then-rerank pipeline, where the quality of the initial retrieval stage critically impacts final ranking performance. Traditional set-based metrics, such as recall, do not capture the nuanced impact of document relevance distributions, leading to a potential misalignment with downstream ranking effectiveness. In this paper, we propose Normalized Set-Level Ideal DCG (nSIDCG), a novel, optimistic metric that normalizes the set-level Ideal DCG against the ground-truth Ideal DCG. This approach ensures a more holistic evaluation of retrieval quality by taking into account both the presence and the importance of relevant documents. Through extensive experiments on TREC Deep Learning Tracks (2019-2023), we demonstrate that nSIDCG exhibits a stronger correlation with the final nDCG scores compared to conventional recall-based metrics, making it a more reliable early stage retrieval assessment tool. Our findings suggest that nSIDCG can serve as an effective metric, helping system designers optimize retrieval models before the computationally expensive reranking phase. Codes: https://github.com/RichardFu123/nsidcg.