Time to judge relevance as an indicator of assessor error
Mark D. Smucker, Chandra Prakash Jethani · 2012
When human assessors judge documents for their relevance to a search topic, it is possible for errors in judging to occur. As part of the analysis of the data collected from a 48 participant user study, we have discovered that when the participants made relevance judgments, the average participant spent more time to make errorful judgments than to make correct judgments. Thus, in relevance assessing scenarios similar to our user study, it may be possible to use the time taken to judge a document as an indicator of assessor error. Such an indicator could be used to identify documents that are candidates for adjudication or reassessment.