Parameter Tuning with User Models
Vinay Deolalikar · 2014
Can we effectively influence aggregate user behavior in a cluster based retrieval (CBR) system by tuning its parameters? This question combines parameter tuning with models of user behavior. To address this question, we propose an approach based on three components: user model, criterion metric, and sensitivity analysis. We then demonstrate this approach on one of the most frequently asked questions to designers and operators of CBR systems in enterprises: namely, "suggest a value for k." Both the users and the system desire a value that is likely to maximize user satisfaction, and sway them towards a cluster based examination of their retrieved result set (rather than prefer the original ranked retrieved list). Based on observed user behavior in CBR systems, we posit a two-stage user model. We isolate its core element, which is a "query coverage metric." We then perform an empirical sensitivity analysis of this metric. Our analysis reveals that this metric is, surprisingly, robust to changes in k (i.e., insensitive to k) in a wide range around its de-facto value. We conclude that in cases where our model approximates user behavior, the system cannot substantially increase the chances of the user resorting to CBR by tuning k. This has practical implications on the design and day-to-day operation of CBR systems. Similar analyses can be carried out for other parameters.