Cooling schedules for learning in neural networks
Tom Heskes, Eddy T. P. Slijpen, Bert Kappen · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1993
We derive cooling schedules for the global optimization of learning in neural networks. We discuss a two-level system with one global and one local minimum. The analysis is extended to systems with many minima. The optimal cooling schedule is (asymptotically) of the form \ensuremath{\eta}(t)=${\mathrm{\ensuremath{\eta}}}^{\mathrm{*}}$/lnt, with \ensuremath{\eta}(t) the learning parameter at time t and ${\mathrm{\ensuremath{\eta}}}^{\mathrm{*}}$ a constant, dependent on the reference learning parameters for the various transitions. In some simple cases, ${\mathrm{\ensuremath{\eta}}}^{\mathrm{*}}$ can be calculated. Simulations confirm the theoretical results.