Emergent scale invariance in neural networks
Mikhail I. Katsnelson, Vitaly Vanchurin, Tom Westerhout · Physica A Statistical Mechanics and its Applications · 2022
We demonstrate, both analytically and numerically, that learning dynamics of neural networks is generically attracted towards a scale-invariant state. The effect can be modeled with quartic interactions between non-trainable variables (e.g. states of neurons) and trainable variables (e.g. weight matrix). Non-trainable variables are rapidly driven towards stochastic equilibrium and trainable variables are slowly driven towards learning equilibrium described by a scale-invariant distribution on a wide range of scales.