Structured Recurrent Temporal Restricted Boltzmann Machines
Roni Mittelman, Benjamin J. Kuipers, Silvio Savarese, Honglak Lee · 2014
The recurrent temporal restricted Boltzmann ma-chine (RTRBM) is a probabilistic time-series model. The topology of the RTRBM graphical model, however, assumes full connectivity be-tween all the pairs of visible units and hidden units, thereby ignoring the dependency structure within the observations. Learning this structure has the potential for not only improving the pre-diction performance, but also revealing impor-tant dependency patterns in the data. For ex-ample, given a meteorological dataset, we could identify regional weather patterns. In this work, we propose a new class of RTRBM, which we refer to as the structured RTRBM (SRTRBM), which explicitly uses a graph to model the de-pendency structure. Our technique is related to methods such as graphical lasso, which are used to learn the topology of Gaussian graphical mod-els. We also develop a spike-and-slab version of the RTRBM, and combine it with the SRTRBM to learn dependency structures in datasets with real-valued observations. Our experimental re-sults using synthetic and real datasets demon-strate that the SRTRBM can significantly im-prove the prediction performance of the RTRBM, particularly when the number of visible units is large and the size of the training set is small. It also reveals the dependency structures underly-ing our benchmark datasets. 1.