Do Batch and User Evaluations Give the Same Results? An Analysis from the TREC-8 Interactive Track.
William Hersh, Andrew H. Turpin, Susan Price, Dale F. Kraemer, Benjamin K S Chan, Lynetta Sacherek, Daniel Olson · Text REtrieval Conference · 1999
An unanswered question in information retrieval research is whether improvements in system performance demonstrated by batch evaluations confer the same benefit for real users. We used the TREC-8 Interactive Track to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline, we used it with real users searching on an instance recall task. Our results showed no improvement; although there was overall average improvement comparable to the batch results, it was not statistically significant and due to the effect of just one out of the six queries. Further analysis with more queries is necessary to resolve this question.