Deterministic Annealing for Stochastic Variational Inference.
Farhan Abrol, Stephan Mandt, Rajesh Ranganath, David M. Blei · arXiv (Cornell University) · 2014
Stochastic variational inference (SVI) maps pos-terior inference in latent variable models to non-convex stochastic optimization. While they en-able approximate posterior inference for many otherwise intractable models, variational infer-ence methods suffer from local optima. We in-troduce deterministic annealing for SVI to over-come this issue. We introduce a temperature parameter that deterministically deforms the ob-jective, and then reduce this parameter over the course of the optimization. Initially it encourages high entropy variational distributions, which we find eases convergence to better optima. We test our method with Latent Dirichlet Allocation on three large corpora. Compared to SVI, we show improved predictive likelihoods on held-out data. 1