Measuring and Comparing the Consistency of IR Models for Query Pairs with Similar and Different Information Needs
Procheta Sen, Sourav Saha, Debasis Ganguly, Manisha Verma, Dwaipayan Roy · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · 2022
A widespread use of supervised ranking models has necessitated an investigation on how consistent their outputs align with user expectations. While a match between the user expectations and system outputs can be sought at different levels of granularity, we study this alignment for search intent transformation across a pair of queries. Specifically, we propose a consistency metric, which for a given pair of queries - one reformulated from the other with at least one term in common, measures if the change in the set of the top-retrieved documents induced by this reformulation is as per a user's expectation. Our experiments led to a number of observations, such as DRMM (an early interaction based IR model) exhibits better alignment with set-level user expectations, whereas transformer-based neural models (e.g., MonoBERT) agree more consistently with the content and rank-based expectations of overlap.