A Subtopic Taxonomy-Aware Framework for Diversity Evaluation.
Fei Chen, Yiqun Liu, Min Zhang, Shaoping Ma, Lei Chen · 2013
To evaluate search result diversification, which is supposed to meet different needs behind a same query, a number of evaluation frameworks are proposed and adopted by benchmarks such as TREC and NTCIR. These frameworks usually do not consider the subtopic taxonomy information. Many previous works on docu-ment ranking have shown that different kinds of information needs require different ranking strategies to be satisfied. It is thus necessary to involve subtopic taxonomy in the evaluation frame-work of search result diversification. In this paper, we propose a novel framework called the Subtopic Taxonomy-Aware (STA) framework to redefine the existing measures. Measures in this new framework take the subtopic taxonomy information into con-sideration for diversity evaluations. On the other hand, finding the optimal diversified results of many measures is proved as a NP-hard problem. We also propose a pruning algorithm which can decrease this problem to a computable search. Experiments based on both the TREC and NTCIR test collections show the effective-ness of our proposed framework.