A Novel Shadow Variable Catcher for Addressing Selection Bias in Recommendation Systems
Qingfeng Chen, Bo-Quan Wei, Debo Cheng, Jiuyong Li, Lin Liu, Shichao Zhang · 2024
Recommender systems rely on observational data to predict user ratings for unseen items. Since the observational data is typically missing not at random (MNAR), they contain biases, predominantly selection bias, thus models trained on such data are inherently biased. If a shadow variable, which is a variable used instead of user's latent variables that influence both the treatment and the outcome, can be identified, it is possible to build unbiased models for recommender systems. To overcome the challenge of manually identifying valid shadow variables in the data, we propose a Shadow Variable Catcher (SVC), a model designed to learn the latent representation of shadow variables from observational data. By using the learned representation of shadow variables by SVC, we develop the Shadow Debiased Recommender (SDR) method to build an unbiased collaborative filtering model for addressing selection bias in recommender systems. Comprehensive experiments on both synthetic and real-world datasets, have verified the performance of SDR and demonstrated its effectiveness and robustness, and offer new insights into the mitigation of bias in recommender systems.