Relevance Judgments
Ziying Yang · 2017
Conventionally, relevance judgments were assessed using ordinal relevance scales such as binary and Sormunen categories [9]. Such judgments record how much overlap there is between the document and the topic. However they have been argued as unreliable and not objective [3, 5, 10] because: (1) documents are usually assessed by limited numbers of experts, with different viewpoints of relevance because of individual factors such as gender, age and background [1]; (2) the distinctions of relevance levels expected by users disparate types may be diverse [7]; (3) assessors' examining criteria drift in varying degrees as more documents are judged [8]; (4) many judgment ties are generated using ordinal scales. In order to have a better understanding of users' perceptions of relevance and collect data with high fidelity, we propose to use the Pairwise Preference technique [2] to collect relevance judgments from a crowdsourcing platform. With the collected judgments, a computed rank list containing all judged documents for each topic will be generated with the goal of having fewer relevance ties.