Differentially-private learning and information theory
Darakhshan Mir · 2012
Using results from PAC-Bayesian bounds in learning theory, we formulate differentially-private learning in an information theoretic framework. This, to our knowledge, is the first such treatment of this increasingly popular notion of data privacy. We examine differential privacy in the PAC-Bayesian framework and through such a treatment examine the relation between differentially-private learning and learning in a scenario where we seek to minimize the expected risk under mutual information constraints. We establish a connection between the exponential mechanism, which is the most general differentially private mechanism and the Gibbs estimator encountered in PAC-Bayesian bounds. We discover that the goal of finding a probability distribution that minimizes the so-called PAC-Bayesian bounds (under certain assumptions), leads to the Gibbs estimator which is differentially-private.