Continuous Evaluation in Information Retrieval Across Methods and Time
Jüri Keller · 2025
Evaluating Information Retrieval (IR) systems is essential yet challenging. The dynamic nature of information and relevance further complicates IR. For example, Adar et al. observed that as early as 2009, websites frequently changed multiple times per hour. In response, recent IR systems have become more personalized, semantic, and context-aware. They evolved from lexical ranking functions to complex neural models embedded in feature-rich systems. While this often improves retrieval quality, it also challenges their reliability and robustness. The proposed research is motivated by the overarching goal of maintaining the trustworthiness of IR systems. To do so a rigorous evaluation is needed. Only by assessing the quality of a system can it be maintained and improved. Such an evaluation can not take place in isolation but must consider the dynamics of the search setting at all stages of a system-from development to maintenance and improvement. Based on the CRISP-DM methodology, these stages are sketched out as an ''IR Life Cycle'', which we will further define and oppose with a continuous evaluation framework in future work.