Continuous Result Delta Evaluation of IR Systems
Gabriela González-Sáez · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · 2022
Classical evaluation of information retrieval systems evaluates a system in a static test collection. In the case of Web search, the evaluation environment (EE) is continuously changing and the hypothesis of using a static test collection is not representative of this changing reality. Moreover, the changes in the evaluation environment, as the document set, the topics set, the relevance judgments, and the chosen metrics, have an impact on the performance measurement [1, 4]. To the best of our knowledge, there is no way to evaluate two versions of a search engine with evolving EEs.