Exploiting user disagreement for web search evaluation
Thomas R. DeMeester, Robin Aly, Djoerd Hiemstra, Dong Nguyen, Dolf Trieschnigg, Chris Develder · 2014
To express a more nuanced notion of relevance as compared to binary judgments, graded relevance levels can be used for the evaluation of search results. Especially in Web search, users strongly prefer top results over less relevant results, and yet they often disagree on which are the top results for a given information need. Whereas previous works have generally considered disagreement as a negative effect, this paper proposes a method to exploit this user disagreement by integrating it into the evaluation procedure.