Summed Weight Neuron Perturbation: An O(N) Improvement Over Weight Perturbation
Barry Flower, Marwan A. Jabri · 1992
The algorithm presented performs gradient descent on the weight space of an Artificial Neural Network (ANN), using a finite difference to approximate the gradient. The method is novel in that it achieves a computational complexity similar to that of Node Perturbation, O(N 3 ), but does not require access to the activity of hidden or internal neurons. This is possible due to a stochastic relation between perturbations at the weights and the neurons of an ANN. The algorithm is also similar to Weight Perturbation in that it is optimal in terms of hardware requirements when used for the training of VLSI implementations of ANN's. 1 INTRODUCTION Optimization of the weights of an ANN may be performed by, the application of a gradient descent technique. The gradient may be calculated directly as in Backpropagation, or it may be approximated by a Finite Difference Method which is what we concern ourselves with in this paper. These methods lend themselves to the task of training hardware imp...