SIMPLIFYING NEURAL NETS BY DISCOVERING FLAT MINIMA
Sepp Hochreiter, Jürgen Schmidhuber · 1994
We present a new algorithm for finding low complexity networks with high generalization capability. The algorithm searches for large connected regions of so-called "flat" minima of the error function. In the weight-space environment of a "flat" minimum, the error remains approximately constant. Using an MDL-based argument, flat minima can be shown to correspond to low expected overfitting. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity. Experiments with feedforward and recurrent nets are described. In an application to stock market prediction, the method outperforms conventional backprop, weight decay, and "optimal brain surgeon". 1 INTRODUCTION Previous algorithms for finding low complexity networks with high generalization capability are based on significant prior assumptions. They can be broadly classified as follows: (1) Assumptions about the prior weight distribution. Hinton and van Camp [3] and Williams [17] ass...