Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification
Junfeng Wen, Chun-Nam John Yu, Russell Greiner · 2014
Many learning situations involve learning the conditional distribution ppy|xq when the training instances are drawn from the training distribu-tion ptrpxq, even though it will later be used to predict for instances drawn from a different test distribution ptepxq. Most current approaches fo-cus on learning how to reweigh the training ex-amples, to make them resemble the test distribu-tion. However, reweighing does not always help, because (we show that) the test error also de-pends on the correctness of the underlying model class. This paper analyses this situation by view-ing the problem of learning under changing dis-tributions as a game between a learner and an ad-versary. We characterize when such reweighing is needed, and also provide an algorithm, robust covariate shift adjustment (RCSA), that provides relevant weights. Our empirical studies, on UCI datasets and a real-world cancer prognostic pre-diction dataset, show that our analysis applies, and that our RCSA works effectively. 1.