What deliberately degrading search quality tells us about discount functions
Paul Thomas, Timothy Willem Jones, David Hawking · 2011
Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data. Categories and Subject Descriptors: H.3.4 [Information