Relevance assessment (un-)reliability in information retrieval
Pavel Metrikov · 2016
Collecting relevance assessments is a very important procedure in Information Retrieval. It is conducted to (1) evaluate the performance of an existing search engine, or (2) build and train a new one. While most of the popular performance evaluation measures and search engine training algorithms assume the relevance assessments are accurate and reliable, in practice this assumption is often violated. Whether intentionally or not, assessors may provide noisy and inconsistent relevance judgments potentially leading to (1) wrong conclusions about performance of a search engine, or (2) inefficient or suboptimal training of a search engine.