On retrieval tests with an inhomogeneous query collection
Friedbert Jochum · 1985
Retrieval tests are often performed with an inhomogeneous query collection, that means certain properties of queries vary within the collection. This paper investigates under which conditions this inhomogeneity may lead to a loss of measurement precision and validity and should be controlled by an experimental design. In particular, interrelationships between the design of a retrieval test, inhomogeneity of the query collection and measurement theoretical properties of the evaluation measure will be discussed. Based on these results, some recommendations for experimental design of retrieval tests are presented.