Application of statistical mechanics in unsupervised learning and associative memory
TianQi Hou · 2021
Neural networks are highly complex dynamical systems consisting of large numbers of neurons interacting through synapses. How can we formulate adequate theoretical frameworks for understanding such systems from statics to dynamics, and from macroscopic to microscopic? In this thesis, we analyse two paradigms, Restricted Boltzmann Machine (RBM) and Hopfield model, which have been studied by tools originating from disordered statistical mechanics. RBM, a two layer neural network to learn the hidden features in datasets and generate data, is the cornerstone of unsupervised learning. Hopfield model is fundamental to theoretical neuroscience, which also can be regarded as a RBM with quadratic function in the hidden layer. In the first part, we analyse the permutation symmetry of RBM with two hidden units, revealing a series of continuous phase transitions driven by data. In the second part, we study a popular correlated Hopfield model which is used as a prototype of many neuroscience experiments and investigate the model from a dynamics perspective using random matrices and its equilibrium properties by the replica theory. From these two angles, we get more insights into the temporal and spatial correlations in neural circuits.