Understanding user situational relevance in ranking web search results
Eugene Opoku-Mensah, Fengli Zhang, Fan Zhou, Philemon Kibiwott Kittur · 2017
Relevance plays a key role in determining how ranking features are weighed in rank algorithms. In particular, the Search Engine Result Page (SERP) presents documents from the algorithmic relevance perspective. User relevance, however, is situational and describes the searcher's perception of the contextual document which is interpreted at rank time. Therefore, without it the monotonic decreasing click trend portrayed by rank algorithms will not hold for real time searchers. Instead, users select low ranked documents over high ones or reject the ranking, leading to frustration and several re-query tasks. To understand users situational relevance (USR), we formulate a decreasing probability function to measure each user's personalized probabilistic relevance (PPR). Next, we find an aggregated PPR representing the USR across several query categories. Then, we apply maximum probabilities swapping to modify the rank function and re-rank documents. Finally, we use Normalized Discounted Cumulative Gain (NDCG) and click weight metrics for evaluation. Our results show that a clear interpretation and incorporation of user situational relevance improves click satisfaction by 12.86%, and optimizes click through rate up to 42%.