On the suitability of diversity metrics for learning-to-rank for diversity
Rodrygo L. T. Santos, Craig Macdonald, Iadh Ounis · 2011
An optimally diverse ranking should achieve the maximum coverage of the aspects underlying an ambiguous or under-specified query, with minimum redundancy with respect to the covered aspects. Although evaluation metrics that re-ward coverage and penalise redundancy provide intuitive ob-jective functions for learning a diverse ranking, it is unclear whether they are the most effective. In this paper, we con-trast the suitability of relevance and diversity metrics as ob-jective functions for learning a diverse ranking. Our results in the context of the diversity task of the TREC 2009 and 2010 Web tracks show that diversity metrics are not neces-sarily better suited for guiding a learning approach. More-over, the suitability of these metrics is compromised as they try to penalise redundancy during the learning process.