Sensitivity of IR systems Evaluation to Topic Difficulty
Koji Eguchi, Kazuko Kuriyama, Noriko Kando · 2002
The difficulty of the topics or queries is one of important factors in evaluating information retrieval (IR) systems.This paper analyzes the differences of system ranking affected by the topic difficulty using a test collection 'NTCIR-1,' which is constructed for evaluating Japanese IR systems and composed of (1) the topics, (2) the document database, and (3) the lists of relevant judgments.Furthermore, this paper defines measures for the various features on the topics, and analyzes the correlation between them, in order to investigate the predictability of the topic difficulty.