Geometry of Early Stopping in Linear Networks
Robert H. Dodier · 1995
A theory of early stopping as applied to linear models is presented. The backpropagation learning algorithm is modeled as gradient descent in continuous time. Given a training set and a validation set, all weight vectors found by early stopping must lie on a certain quadric surface, usually an ellipsoid. Given a training set and a candidate early stopping weight vector, all validation sets have least-squares weights lying on a certain plane. This latter fact can be exploited to estimate the probability of stopping at any given point along the trajectory from the initial weight vector to the leastsquares weights derived from the training set, and to estimate the probability that training goes on indefinitely. The prospects for extending this theory to nonlinear models are discussed. 1 INTRODUCTION `Early stopping' is the following training procedure: Split the available data into a training set and a "validation" set. Start with initial weights close to zero. Apply gradient descent (ba...