Inconsistent Ranking Assumptions in Medical Search and Their Downstream Consequences
Daniel Cohen, Kevin Du, Bhaskar Mitra, Laura Y. Mercurio, Navid Rekabsaz, Carsten Eickhoff · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · 2022
Given a query, neural retrieval models predict point estimates of relevance for each document; however, a significant drawback of relying solely on point estimates is that they contain no indication of the model's confidence in its predictions. Despite this lack of information, downstream methods such as reranking, cutoff prediction, and none-of-the-above classification are still able to learn effective functions to accomplish their respective tasks. Unfortunately, these downstream methods can suffer poor performance when the initial ranking model loses confidence in its score predictions. This becomes increasingly important in high-stakes settings, such as medical searches that can influence health decision making.