Including summaries in system evaluation
Andrew H. Turpin, Falk Scholer, Kalvero Järvelin, Mingfang Wu, J. Shane Culpepper · 2009
In batch evaluation of retrieval systems, performance is calculated based on predetermined relevance judgements applied to a list of documents returned by the system for a query. This evaluation paradigm, however, ignores the current standard operation of search systems which require the user to view summaries of documents prior to reading the documents themselves.