Decision-Making with Unknown Priors in Supervised Classication
Saerens Marco, Christine Decaestecker · 2010
In this work, we examine minimum expected cost decision making in the presence of uncertainty about the class priors. More precisely, a classier is trained on a training set and, once the parameters are estimated, it is applied on new real-world data that have to be labeled. We then examine the situation in which the a priori probabilities of the classes (priors) in the real-world data sets are unknown and are suspected to be dierent from those encountered in the training set (new sampling conditions), while the within-class densities remain unchanged. Various scenarios are considered, according to whether estimates of the new priors are available, and are discussed from a decision-making point of view, aiming to optimize the expected cost in the new sampling conditions. In particular, we show that when no information at all is available about the priors of the data sets on which the classier will be applied, the optimal decision rule is based on the likelihood; that is, equal priors for all classes. This helps to provide a clear interpretation to the \rule of thumb that is usually applied in this situation: to train the classier with equal proportions of observations from each class (see, e.g., [30]). The analysis method is quite generic and could easily be adapted to various other situations.