TREC 2005 Robust Track Experiments Using PIRCS.
Kui Lam Kwok, Laszlo Grunfeld, Norbert Dinstl, Peter Shaohua Deng · 2005
There were two sub-tasks in the TREC2004 Robust track: given a set of topics, a) improve the effectiveness of the lowest performing 25%, and b) predict their ranking according to their average precision. For task a), we followed the strategy introduced by us last year to improve ad-hoc retrieval by employing the web as an external thesaurus to supplement a given topic description. A new method of probing the web based on a given topic statement called ‘window rotation’ was tested. For task b) we employed e-SVR (epsilon support vector regression) to predict performance of test topics based on training with some simple features such as document frequencies, query term frequencies. This allows performance prediction without retrieval. Features were also added from a retrieval list with the hope that they may predict later stage or web-assisted retrieval better. 200 old topics were used for training to predict the ranking of 49 new topics, as well as the whole set of 249. Runs were done that made use of title only, description only section of a topic, and titledescription-combination retrieval lists. Ten submissions including runs that were based on initial retrieval only, retrievals with pseudo-relevance feedback, and with web-assistance. Evaluation shows that we have achieved very good performance for most of our runs. 2 Robust Track – Improving Low Performing Topics 2.1 Background We introduced a new strategy of improving ad-hoc retrieval based on web-assistance in the Robust Track of TREC2003. In initial retrieval, some queries have low average precision performance (weak or hard queries) while others return good values (strong or easy queries). The objective of this track is to automatically improve the effectiveness of weak topics, and others in general. Strong topics can generally be further improved with pseudo-relevance feedback (PRF), but this does not work for weak topics because for them, an initial retrieval would not bring in much useful material for feedback use. One may try to enrich weak topic wordings via a thesaurus to improve term variety, and thereby enhancing initial retrieval results. However choosing an available and appropriate thesaurus of the right domain without prior knowledge of a topic is quite a challenge. We demonstrated in TREC2003 that employing the WWW as an alldomain word-association resource with appropriate filtering can be successful for this Robust Track objective.