Averaging over networks: properties, evaluation, and minimization
Vicente Ruiz de Angulo, Carme Torras · 1998
The expectation of a function of random variables is modelled as the value of the function in the mean value of the variables plus a penalty term. This penalty term is calculated exactly, and the properties of different approximations are analyzed. In particular, for quadratic functions, the penalty term is shown to have a supervised and an unsupervised part. Then, two algorithms for minimizing the expected error of a feedforward network of random weights are compared. One of them is stochastical, while the other is deterministic and more flexible. Given a particular feedforward network architecture and a training set, the deterministic algorithm here presented accurately finds the weight configuration that makes the network response most resistant to a class of weight perturbations. Finally, the study of the most stable configurations of a network unravels some undesirable properties of networks with asymmetric activation functions