Statistical mechanics of EKF learning in neural networks
Bernhard Schottky, David Saad · Journal of Physics A Mathematical and General · 1999
We formulate a learning algorithm for online learning in neural networks using the extended Kalman filter approach, providing a principled and practicable approximation to the full Bayesian treatment. The latter, which constitutes optimal learning, does not require artificial setting of training parameters and allows for the estimation of a wide range of quantities of interest. We analyse the performance of the algorithm using tools of statistical physics in several scenarios: we look at drifting rules represented by linear and nonlinear perceptrons and investigate how different prior settings affect the generalization performance as well as learnability itself. We investigate the learning behaviour of stationary two-layer network, where the algorithm seems to avoid the, otherwise common, problem of long symmetric plateaus.