On the Representational Efficiency of Restricted Boltzmann Machines
James Martens, Arkadev Chattopadhya, Toniann Pitassi, Richard S. Zemel · 2013
This paper examines the question: What kinds of distributions can be efficiently represented by Restricted Boltzmann Machines (RBMs)? We characterize the RBM’s unnormalized log-likelihood function as a type of neural network, and through a series of simulation results relate these networks to ones whose repre-sentational properties are better understood. We show the surprising result that RBMs can efficiently capture any distribution whose density depends on the num-ber of 1’s in their input. We also provide the first known example of a particular type of distribution that provably cannot be efficiently represented by an RBM, as-suming a realistic exponential upper bound on the weights. By formally demon-strating that a relatively simple distribution cannot be represented efficiently by an RBM our results provide a new rigorous justification for the use of potentially more expressive generative models, such as deeper ones. 1