Transient dynamics of on-line learning in two-layered neural networks
Michael L. Biehl, Peter Riegler, Christian Wöhler · Journal of Physics A Mathematical and General · 1996
The dynamics of on-line learning in neural networks with continuous units is dominated by plateaux in the time dependence of the generalization error. Using tools from statistical mechanics, we show for a soft committee machine the existence of several fixed points of the dynamics of learning that give rise to complicated behaviour, such as cascade-like runs through different plateaux with a decreasing value of the corresponding generalization error. We find learning-rate-dependent phenomena, such as splitting and disappearing of fixed points of the equations of motion. The dependence of plateau lengths on the initial conditions is described analytically and simulations confirm the results.