Generalization Error and the Expected Network Complexity
Chuanyi Ji · 1993
For two layer networks with n sigmoidal hidden units, the generalization error is shown to be bounded by O(E~) O ( (EK)d l N) K + N og, where d and N are the input dimension and the number of training samples, re-spectively. E represents the expectation on random number K of hidden units (1:::; I \\:::; n). The probability Pr(I { = k) (1:::; k:::; n) is (kt.erl11ined by a prior distribution of weights, which corresponds to a Gibbs distribtt! ion of a regularizeI'. This relationship makes it possible to characterize explicitly how a regularization term affects bias/variance of networks. The bound can be obtained analytically for a large class of commonly used priors. It can also be applied to estimate the expected net.work complexity Ef { in practice. The result provides a quantitative explanation on how large networks can generalize well. 1