New Performance Metrics Based on Multigrade Relevance: Their Application to Question Answering
Tetsuya Sakai · NTCIR · 2004
This paper proposes two new InformationRetrieval performance metrics based on multigrade relevance, called Q-measure and R-measure, which are akin to Cumulative Gain and Average Weighted Precision but are arguably more reliable. We then show how Qmeasure can be appliedto Question Answering involving ranked lists of exact answers, and discuss its advantages over Reciprocal Rank through an experiment using the QAC1 test collection. The appendices of this paper contain theorem proofs concerning Q-measure and R-measure, as well as a study of Q-measure and R-measure as Information Retrieval evaluation metrics using the runs submitted to the NTCIR-3 CLIR task. We plan to conduct similar experiments for the NTCIR QAC tasks using Q-measure as a Question Answering evaluation metric, if the QAC submissionfiles become available for research purposes.