Nugget-Based Computation of Graded Relevance
Charles L. A. Clarke · 2013
We propose a simple method for assigning graded relevance values to documents judged during the course of a retrieval experiment. In making this proposal, we aim to avoid the potential for ambiguity and greater cognitive load associated with standard graded relevance judgments. Under our pro-posal, we first decompose a retrieval topic into a number of informational nuggets. For each document, a binary judg-ment is made with respect to each nugget. The ratio of rele-vant nuggets to total nuggets becomes the graded relevance value assigned to that document. To provide support for this idea, we turn to test collections created for the TREC Web Track. Along with the usual graded relevance judg-ments required by traditional effectiveness measures, these test collections include topic decompositions created for the purpose of evaluating novelty and diversity. By exploiting these test collections for our own purposes, we demonstrate a clear relationship between our proposed method and tradi-tional graded relevance. In addition to supporting our pro-posal, our experiments suggest that informational nuggets can provide a unified approach to relevance assessment, sup-porting both traditional effectiveness measures and newer measures of novelty and diversity.