Risk-Reward Trade-offs in Rank Fusion
Rodger Benham, J. Shane Culpepper · 2017
Rank fusion is a powerful technique that merges multiple system runs to produce a single top-k list that often has much higher effectiveness than any single system can produce. Recently, there has been renewed interest in rank fusion in the IR community as these techniques can also be combined with query variations to produce highly effective runs. In this work, we comprehensively evaluate several state-of-the-art fusion algorithms in the context of risk. Like many re-ranking algorithms, there is a risk-reward trade-off in rank fusion, where improving the retrieval effectiveness for most queries often comes at the expense of others. Since system performance is usually compared using only aggregate scores for an evaluation metric, the risk is potentially obscured. In this work, we explore the use of the risk-based evaluation metrics over deep and shallow evaluation goals, and show that the risk-reward payoff in keyword queries can in fact be significantly improved when careful combinations of system and query variations are fused into a single run.