Incorporating Efficiency in Evaluation
Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov · 2013
The trade-off between retrieval efficiency and effectiveness naturally arises in various domains of information retrieval research. In this paper, we argue that the users ’ reaction to the system efficiency should be incorporated into the cascade model of the user behaviour and that an effectiveness-aware modification of a state-of-the-art evaluation metric can be derived. We propose to base the study on the anonymised browser/toolbar log data, and our preliminary experiments demonstrate that the users ’ reaction to the delays observed in the dataset is consistent with one reported previously. 1.