Annealing paths for the evaluation of topic models
James R. Foulds, Padhraic Smyth · 2014
Statistical topic models such as latent Dirich-let allocation have become enormously popu-lar in the past decade, with dozens of learning algorithms and extensions being proposed each year. As these models and algorithms continue to be developed, it becomes increasingly impor-tant to evaluate them relative to previous tech-niques. However, evaluating the predictive per-formance of a topic model is a computationally difficult task. Annealed importance sampling (AIS), a Monte Carlo technique which operates by annealing between two distributions, has pre-viously been successfully used for topic model evaluation (Wallach et al., 2009b). This tech-nique estimates the likelihood of a held-out doc-ument by simulating an annealing process from the prior to the posterior for the latent topic as-signments, and using this simulation as an im-portance sampling proposal distribution. In this paper we introduce new AIS annealing paths which instead anneal from one topic model to another, thereby estimating the relative perfor-mance of the models. This strategy can exhibit much lower empirical variance than previous ap-proaches, facilitating reliable per-document com-parisons of topic models. We then show how to use these paths to evaluate the predictive perfor-mance of topic model learning algorithms by effi-ciently estimating the likelihood at each iteration of the training procedure. The proposed method achieves better held-out likelihood estimates for this task than previous algorithms with, in some cases, an order of magnitude less computation. 1