Effect of Intent Descriptions on Retrieval Evaluation
Emine Yılmaz, Evangelos Kanoulas, Nick Craswell · 2014
Test collections play an important role in adhoc and diversity retrieval evaluation. Constructing a test collection for adhoc evaluation involves (1) selecting a set of queries to be judged, (2) selecting an intent (topic) description for that query, and (3) obtaining relevance judgments with respect to the specific intent description for that particular query. Recent work showed that the selection of intents play an important role in the relative performance of retrieval systems for diversity evaluation. However, no previous work has analysed how the choice of a specific intent description may affect adhoc evaluation. We show that intent descriptions have a significant impact in adhoc evaluation and that special care should be given as to how the intent descriptions are selected. We further show that it is better to have very general intent descriptions or no intent descriptions at all when constructing test collections for adhoc evaluation. We then focus on diversity evaluation and identify the effect intent descriptions have on diversity based retrieval evaluation. We quantify this effect and discuss experimental design decisions for the optimal distribution of judgment effort across different intents for a query vs. different queries.