Using ancillary statistics in on-line learning algorithms
Huaiyu Zhu, Richard Rohwer · Aston Publications Explorer (Aston University) · 1996
Neural networks are usually curved statistical models. They do not have finite dimensional sufficient statistics, so on-line learning on the model itself inevitably loses information. In this paper we propose a new scheme for training curved models, inspired by the ideas of ancillary statistics and adaptive critics. At each point estimate an auxiliary flat model (exponential family) is built to locally accommodate both the usual statistic (tangent to the model) and an ancillary statistic (normal to the model). The auxiliary model plays a role in determining credit assignment analogous to that played by an adaptive critic in solving temporal problems. The method is illustrated with the Cauchy model and the algorithm is proved to be asymptotically efficient. 1 Introduction Neural network (NN) training algorithms are essentially statistical estimators since they map random samples to some general rules or distributions underlying these samples. The main difference between NN models and c...