Information retrieval evaluation with humans in the loop
Gabriella Kazai · 2014
The evaluation and tuning of information retrieval (IR) systems based on the Cranfield paradigm requires purpose built test collections, which include sets of human contributed relevance labels, indicating the relevance of search results to a set of user queries. Traditional methods of collecting relevance labels rely on a fixed group of hired expert judges, who are trained to interpret user queries as accurately as possible and label documents accordingly. Human judges and the obtained relevance labels thus provide a critical link within the Cranfield style IR evaluation framework, where disagreement among judges and the impact of variable judgment sets on the final outcome of an evaluation is a well studied issue. There is also reported evidence that experiment outcomes can be affected by changes to the judging guidelines or changes in the judge population.