Random Energy Models, Optimal Learning Machines and Beyond
Matteo Marsili · arXiv (Cornell University) · 2019
I consider an ensemble of optimisation problems over random functions of many variables, part of which describe a sub-system and the rest account for its interaction with the environment. The function being optimised is drawn from a stretched exponential distribution, with exponent $\gamma$, independently for each configuration of the whole system. A well defined thermodynamic limit is attained as the size of the sub-system and of the environment diverge, with a fixed ratio, recovering the physics of the Random Energy Model (REM). This is characterised by a freezing phase transition which is continuous for $\gamma>1$ and discontinuous for $\gamma 1$ corresponds to learnable energy landscape for which the behaviour of the sub-system becomes more predictable as the size of the environment increases. For $\gamma<1$, instead, the energy landscape is ``unlearnable`` and becomes more and more unpredictable as the size of the environment increases. Optimal learning machines, which feature an exponential distribution of energy level ($\gamma=1$), sit at the boundary between these two regions and feature a freezing transition independent of the relative size of the environment. This is consistent with the expectation that efficient representations should be independent of the addition of further irrelevant details.