Towards Reproducible Machine Learning Research in Information Retrieval
Ana Lučić, Maurits Bleeker, Maarten de Rijke, Koustuv Sinha, Sami Jullien, Robert Stojnic · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · 2022
While recent progress in the field of machine learning (ML) and information retrieval (IR) has been significant, the reproducibility of these cutting-edge results is often lacking, with many submissions failing to provide the necessary information in order to ensure subsequent reproducibility. Despite the introduction of self-check mechanisms before submission (such as the Reproducibility Checklist, criteria for evaluating reproducibility during reviewing at several major conferences, artifact review and badging framework, and dedicated reproducibility tracks and challenges at major IR conferences, the motivation for executing reproducible research is lacking in the broader information community. We propose this tutorial as a gentle introduction to help ensure reproducible research in IR, with a specific emphasis on ML aspects of IR research.