Functional optimization of online algorithms in multilayer neural networks
Renato Vicente, Nestor Caticha · Journal of Physics A Mathematical and General · 1997
We study the online dynamics of learning in fully connected soft committee machines in the student - teacher scenario. The locally optimal modulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximize the average generalization error decay per example. Simulations results for the resulting algorithm are presented for a few cases. The symmetric phase plateaux are found to be vastly reduced in comparison to those found when online backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given.