High-temperature Expansions for Learning Models of Nonnegative Data
Oliver B. Downs · 2000
Recent work has exploited boundedness of data for learning new types of generative model. For nonnegative data it has been shown that the maximum-entropy generative model is a Nonnegative Boltzmann Distribution not a Gaussian distribution, when the model is constrained to match the first and second order statistics of the data. Learning for practical sized problems is made difficult by computing expectations under this distribution. Here I present a second-order approximation for the model, obtained using a "high-temperature" expansion. The result is analogous to the TAP-Onsager equations for the Ising model. The theory is tested on learning a bimodal 2-dimensional model, and in learning a high-dimensional translationally-invariant distribution. Introduction Unsupervised learning of generative and feature-extracting models for continuous nonnegative data has recently been proposed [1], [2]. In [1], it was pointed out that the maximum entropy distribution (matching 1st- and 2nd-order ...