Deep Tempering with Nested Restricted Boltzmann Machines
Clément Roussel, Jorge Fernández-de-Cossio-Díaz, Simona Maria Cocco, Rémi Monasson · HAL (Le Centre pour la Communication Scientifique Directe) · 2023
Distributions of high-dimensional data can be learnt with unsupervised architectures, such as restricted Boltzmann machines (RBM). However, the resulting models are often uneasy to sample when the data distributions include multiple modes. We here consider deep tempering, a paralleltempering-like Monte Carlo sampling algorithm based on a chain of several restricted Boltzmann machines (RBM), where hidden configuration of a machine can be exchanged with the visible configurations of the next one along the chain. Replica exchanges between the different RBM is facilitated by the increasingly clustered representations learnt by deeper RBMs along the chain, allowing for fast transitions between the different modes of the data distribution. We explain why deep tempering works on hierarchical data, and introduce a theoretical framework to understand how hyperparameters, such as the aspect ratios of the RBMs and the weight regularization should be chosen. Our findings are illustrated on two datasets: MNIST and in silico Lattice Proteins.