On information divergence measures, surrogate loss functions and decentralized hypothesis testing
XuanLong Nguyen, Martin J. Wainwright, Michael I. Jordan · 2005
We establish a general correspondence between two classes of statistical functions: Ali-Silvey distances (also known as f-divergences) and surrogate loss functions. Ali-Silvey distances play an important role in signal processing and information theory, for instance as error exponents in hypothesis testing problems. Surrogate loss functions (e.g., hinge loss, exponential loss) are the basis of recent advances in statistical learning methods for classification (e.g., the support vector machine, AdaBoost). We provide a connection between these two lines of research, showing how to determine the unique f-divergence induced by a given surrogate loss, and characterizing all surrogate loss functions that realize a given f-divergence. The correspondence between f-divergences and surrogate loss functions has applications to the problem of designing quantization rules for decentralized hypothesis testing in the framework of statistical learning (i.e., when the underlying distributions are unknown, but the learner has access to labeled samples). 1