C-TEST: Supporting novelty and diversity in testfiles for search tuning
David Hawking, Thomas Rowlands, Paul Thomas · 2009
Tuning a search facility such as a Web search engine, or an enterprise search tool deployed in a particular organisation, is an economically important activity. Intuitively, an important end goal of tuning should be to maximise satisfaction across the searchers who will use the facility. Tuning should therefore use an unbiased sample of actual search requests, a judging process accurately modelling that of real searchers, and measures optimally correlated with real satisfaction. The process must reflect that the result set for a particular query may be judged by submitters with different underlying information needs. It should also model users ’ impatience with duplicate results. Here, we describe a formally-defined, practical, public-domain testfile format suitable for use in batch tuning but capable of reflecting these important aspects. We hope that the formats will soon be supported by all major academic IR systems. DTDs for test and result files, plus an associated toolkit and C-TEST example testfiles for some TREC tasks, are available at es.csiro.au/C-TEST.