User-oriented evaluation methods for information retrieval: a case study based on conceptual models for query expansion
Jaana Kekäläinen, Kalervo Järvelin · 2003
This paper discusses evaluation methods based on the use of non-dichotomous relevance judgements in information retrieval (IR) experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is deskable from the user's point of view in modem large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulated gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. Query expansion is based on concepts, which are selected from a conceptual model, and then expanded by semantic relationships given in the model. The test is run with a best match retrieval system (inQuery) in a text database consisting of newspaper articles.