A qualitative exploration of secondary assessor relevance judging behavior
Aiman L. Al-Harbi, Mark D. Smucker · 2014
Secondary assessors frequently differ in their relevance judgments. Primary assessors are those that originate a search topic and whose judgments truly reflect the assessor's relevance criteria. Secondary assessors do not originate the search and must instead attempt to make relevance judgments based on a description of what is and is not relevant. Secondary assessors may be hired to help in the construction of test collections. Currently our knowledge about secondary assessors is largely limited to quantitative measurements of the differences between judgments produced by secondary and primary assessors. In order to better understand the behavior of secondary assessors, we conducted a think-aloud study of secondary assessing behavior. We asked secondary assessors to think-aloud their thoughts as they judged documents. The think-aloud method gives us insight into how relevance decisions are made. We found that assessors are not always certain in their judgments. In the extreme, secondary assessors are forced to make guesses concerning the relevance of documents. We present many reasons and examples of why secondary assessors produce differing relevance judgments. These differences result from the interactions between the search topic, the secondary assessor, the document being judged, and can even apparently be caused by a primary assessor's error in judging relevance. To improve the quality of secondary assessor judgments, we recommend that relevance assessing systems allow for the collection of assessor's certainty and provide a means to help assessors efficiently express their judgment rationale.