The structure of neural network error surfaces
Len Hamey · 1995
Analysis of the error surfaces of feed-forward neural networks is complicated by the high dimensionality of the weight space. Visualisation over one- and twodimensional slices, and Monte Carlo analysis of stationary points can produce misleading results. We show that, in some situations, important features of the error surface can only be visualised by considering the error over non-planar manifolds of weight space. We also show that Monte Carlo simulations can depend critically upon the random step size chosen. The relationship can reveal key properties of the local structure of the error surface. 1 Introduction Apart from the learning algorithm itself, the structure of the error surface of a feed-forward neural network is the single most significant factor in determining the learning behaviour. The presence of local minima may entrap a learning algorithm in a suboptimal solution [5]. In the event of apparent entrapment, a researcher may employ visualisation techniques [2, 3] or Mont...