Differentiaalisesti yksityinen approksimatiivinen Bayes-päättely todennäköisyysmalleille
Joonas Jälkö · Aaltodoc (Aalto University) · 2023
Learning population level characteristics from a set of individuals, belonging to the said population, is the typical aim of statistical inference. When the inference is based on confidential data, measures should be taken to make sure that sensitive data of none of the individuals can be deduced from the results of the statistical inference. In this thesis, I study approximate Bayesian inference under the strict privacy constraint of differential privacy (DP), which allows inferring many of the key features from the data while limiting the effect each individual has to the results, thus keeping the contribution to the analysis a secret. In this thesis, I focus on two families of approximate Bayesian inference methods: variational inference (VI) and Markov chain Monte Carlo (MCMC). Both of these families of methods are applicable for a wide variety of probabilistic models and are widely applied in practice. However, these methods rely on individuals' data through the log-likelihood computation, thus creating a possible channel of privacy leakage. I demonstrate that for a stochastic gradient based VI algorithm, the privacy leakage can be limited with minor modifications to the algorithm that guarantee DP. For a specific type of MCMC algorithm we can have an even more striking result: the algorithm itself guarantees DP as long as the log-likelihood satisfies certain smoothness conditions. This property follows from carefully analyzing the noise arising from the stochasticity of the MCMC method. I show that this noise is enough to limit the individual sample's effect on the results and to guarantee DP. Finally, I study an important application of the DP probabilistic inference: producing privacy-preserving synthetic data. Privacy-preserving synthetic data is typically a data set drawn from a generative model trained under DP. Probabilistic models can be seen as a set of instructions for generating data. Therefore, the probabilistic models trained under DP with the aforementioned DP approximate Bayesian inference techniques can be used to produce DP synthetic data sets. I demonstrate how equipping the probabilistic models with prior information about the data generating process can drastically improve the downstream utility of the synthetic data without compromising the privacy further.