Learning Curves, Model Selection and Complexity of Neural Networks
Noboru Murata, Shuji Yoshizawa, Шун-ичи Амари · 1992
Learning curves show how a neural network is improved as the number of training examples increases and how it is related to the network complexity. The present paper clarifies asymptotic properties and their relation of two learning curves, one concerning the predictive loss or generalization loss and the other the training loss. The result gives a natural definition of the complexity of a neural network. Moreover, it provides a new criterion of model selection. 1 INTRODUCTION The learning curve shows how well the behavior of a neural network is improved as the number of training examples increases and how it is related with the complexity of neural networks. This provides us with a criterion for choosing an adequate network in relation to the number of training examples. Some researchers have attacked this problem by using statistical mechanical methods (see Levin et al. [1990], Seung et al. [1991], etc.) and some by information theory and algorithmic methods (see Baum and Haussler ...