Million Query Track 2007 Overview
James W Allan, Ben Carterette, Javed Aslam, Virgil Pavlu, Blagovest Dachev, Evangelos Kanoulas · 2007
The Million Query (1MQ) track ran for the second time in TREC 2008.The track is designed to serve two purposes: first, it is an exploration of ad-hoc retrieval over a large set of queries and a large collection of documents; second, it investigates questions of system evaluation, in particular whether it is better to evaluate using many shallow judgments or fewer thorough judgments.As with the 2007 track [ACA + 07], participants ran 10,000 queries against a collection of 25 million documents.The 2008 track differed in the following ways:1. Queries were assigned to one of four categories.2. Each query was assigned a target of 8, 16, 32, 64, or 128 judgments.3. Assessors could judge documents "not relevant but reasonable".