How much more is better?: characterizing the effects of adding more IR systems to a combination
Christopher C. Vogt · 2000
We present the results of some expansion experiments for solving the routing, data fusion problem using TREC5 systems. The experiments address the question "How much more is better?" when combining the results of multiple information retrieval systems using a linear combination (weighted sum) model. By investigating all 2way, 3-way, 4-way and 10-way combinations of 10 IR systems on 10 queries, we show that: (1) one can expect potentially significant amounts of improvement in performance over the best system used in the combination if enough systems are used, (2) for this number of candidate systems, the point of diminishing returns is reached when around four systems are used in the combination, (3) queries generally have too few relevant documents, causing little correlation in performance between the training set and test set; thus making it difficult to get test set improvement even when multiple systems are used, and (4) if one knows the relative past performance of the candidate s...