Are Popular Documents More Likely To Be Relevant? A Dive into the ACLIA IR4QA Pools
Tetsuya Sakai, Noriko Kando · 2008
The ACLIA IR4QA Task at NTCIR-7 is an ad hoc document retrieval task involving three document languages. Although IR4QA used pooling for collecting relevance assessments, it was unique in that the pooled documents were sorted before presenting them to the assessors, based on the assumption that “popular” documents are more likely to be relevant than others. We show that this assumption is indeed valid for the IR4QA test collections.