Simulations as a Means to Address Some Limitations of Laboratorybased IR Evaluation
Heikki Keskustalo, Kalervo Järvelin · Tampere University Institutional Repository (Tampere University) · 2010
We suggest using simulations to address some of the limitations of test collection based IR evaluation. In the present paper we explore the effectiveness of short query sessions based on a graphbased view of the searching situation where potential queries (query key combinations) constitute the vertexes of a graph G describing each topic. Session are rules which determine the acceptable query reformulations. Query reformulations manifest as edges in G, and they express the allowed transitions between the vertexes. Multiplequery topical s essions m anifest a s p aths i n G. W e p resent a n e xample o f t his a pproach a ssuming session strategies based on limited query modifications (additions, deletions, or substitutions of few query words). We end by discussing the significance of our approach for IR evaluation.