A Simple Method for Estimating the Weight Decay Parameter
Thorsteinn Rögnvaldsson · 1996
A new method for determining the optimal weight decay parameter is suggested. The method uses early stopping and the estimate = krE 0 (W es )k krR(W es )k ; (1) where W es is the set of weights at the early stopping point, E 0 (W ) is the training data fit error, and R(W ) is the weight decay cost. We demonstrate the method on a large set of data sets, both synthetic and real-life, and show that it is essentially as good an estimator for the optimal as the standard search method. This new method is several orders of magnitude faster than the standard method. 1 Introduction Any practical regression problem is ill-posed (Tikhonov & Arsenin 1977), in the sense that there are infinitely many functions that can fit a finite set of training data perfectly. Furthermore, since any real-life data set consists of noisy inputs and/or outputs, a perfect fit is not even desirable, and models have to be constructed on the basis of some other criteria than just fitting the training data. Me...