Reliable information retrieval evaluation with incomplete and biased judgements
Stefan Büttcher, Charles L. A. Clarke, Peter C. K. Yeung, Ian M. Soboroff · 2007
Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to the pool of judged documents. This may distort the results obtained about the relative quality of the systems evaluated and thus lead to incorrect conclusions about the performance of a particular ranking technique.