Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking
Qing Zhang, Alex Deng, Michelle Du, Huiji Gao, Liwei He, Sanjeev Katariya · 2025
Evaluation plays a crucial role in the development of ranking algorithms on search and recommender systems.It enables online platforms to create user-friendly features that drive commercial success in a steady and effective manner.The online environment is particularly conducive to applying causal inference techniques, such as randomized controlled experiments (known as A/B test), which are often more challenging to implement in fields like medicine and public policy.However, businesses face unique challenges when it comes to effective A/B test.Specifically, achieving sufficient statistical power for conversion-based metrics can be time-consuming, especially for significant purchases like booking accommodations.While offline evaluations are quicker and more cost-effective, they often lack accuracy and are inadequate for selecting candidates for A/B test.To address these challenges, we developed interleaving and counterfactual evaluation methods to facilitate rapid online assessments for identifying the most promising candidates for A/B tests.Our approach not only increased the sensitivity of experiments by a factor of up to 100 (depending on the approach and metrics) compared to traditional A/B testing but also streamlined the experimental process.The practical insights gained from usage in production can also benefit organizations with similar interests.