Feed forward networks and the Cramer-Rao bound
W.F. Schmidt, Robert P. W. Duin · 2003
The weight space of feedforward networks is described by a probability density function where the probability is maximum for the optimal set of weights. This probability density function is given by a property of maximum likelihood estimators and the covariance matrix of this distribution is the Cramer-Rao lower bound. For certain classes of problems the optimization of the mean squared error is equal to the maximum likelihood estimator. For these problems the probability density function is closely related to the mean squared error criterion and therefore results derived from the probability density function hold for the mean squared error surface. An analysis of the probability density function provides some theoretical understanding of the error surface and learning dynamics.>