The Neural Autoregressive Distribution Estimator
Hugo Larochelle, Iain Murray · Edinburgh Research Explorer (University of Edinburgh) · 2011
We describe a new approach for modeling the distribution of high-dimensional vectors of dis-crete variables. This model is inspired by the restricted Boltzmann machine (RBM), which has been shown to be a powerful model of such distributions. However, an RBM typi-cally does not provide a tractable distribution estimator, since evaluating the probability it assigns to some given observation requires the computation of the so-called partition func-tion, which itself is intractable for RBMs of even moderate size. Our model circumvents this difficulty by decomposing the joint dis-tribution of observations into tractable condi-tional distributions and modeling each condi-tional using a non-linear function similar to a conditional of an RBM. Our model can also be interpreted as an autoencoder wired such that its output can be used to assign valid probabilities to observations. We show that this new model outperforms other multivari-ate binary distribution estimators on several datasets and performs similarly to a large (but intractable) RBM. 1