An analysis of coarse-grain parallel training of a neural net
Louis Coetzee, Elizabeth C. Botha · Network Computation in Neural Systems · 1995
In modern day pattern recognition, neural nets are used extensively. General use of a feedforward neural net consists of a training phase followed by a classification phase. Classification of an unknown test vector is very fast and only consists of the propagation of the test vector through the neural net. Training involves an optimization procedure and is very time consuming since a feasible local minimum is sought in weight space. If the training algorithm is based on error backpropagation the optimization procedure consists of the following steps: computation of the activation of the net when all the training examples are presented to it;computation of an error function based on the activation;computation of the gradients at a point in weight space; and finally,the adaptation of the weight values of the net. In this paper we present an analysis of a parallel implementation of the backpropagation algorithm using conjugate-gradient optimization for a three-layered, feedforward neural network, using networked workstations as a virtual parallel machine. The instance of the virtual machine is the PVM system, developed at Oak Ridge National Laboratory. We compare the overall performance of the parallel machine with averaged sequential runs in a typical research environment. From this, we identify the general requirements such as the size of the data set and neural net which render the parallel implementation useful, compared with the sequential execution of the same neural net training procedure.