Improving Collection Selection with Overlap-Awareness
Matthias Bender, Sebastian Michel, Peter Triantafillou, Gerhard Weikum, Christian Zimmer, Ricardo A. Baeza-Yates, Nívio Ziviani, Gary Marchionini, Alistair Moffat, John Irving Tait · Max Planck Institute for Plasma Physics · 2005
Collection selection has been a research issue for years. Most of the existing literature estimates the expected result quality of a collection, typically using precomputed statistics, and ranks the collections accordingly. We believe that this is insufficient if the collections overlap, e.g., in the scenario of autonomous peers crawling the web. We argue for the extension of existing quality measures using estimators of mutual overlap among collections and present experiments in which this combination outperforms CORI, a popular approach based on quality estimation. In our experiments, we use a prototype implementation of a P2P web search engine that allows handling large amounts of data in a distributed and self-organizing manner. Taking overlap into account during collection selection in this scenario can drastically decrease the number of collections that have to be contacted in order to reach a satisfactory level of recall, which is a great step towards the feasibility of distributed web search.