Tutorial on User Simulation for Evaluating Information Access Systems on the Web
Krisztian Balog, ChengXiang Zhai · 2024
Users routinely interact with the Web via information access systems such as search engines and recommender systems. How to accurately evaluate such interactive systems with reproducible experiments is an important, yet difficult challenge. To address this challenge, user simulation has emerged as a promising solution. This half-day tutorial focuses on providing a thorough introduction to user simulation techniques designed specifically for evaluating information access systems on the Web. We systematically review major research progress, covering both general frameworks for designing user simulators, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. We also highlight some important future research directions.