A parallel implementation of the batch backpropagation training of neural networks
Alexander Novokhodko, Stephanie Valentine · 2002
Neural networks, being naturally parallel, inspire researchers to seek efficient implementations for various parallel architectures. However, the vast fine-grain parallelism of many tightly connected simple nodes poses a problem for traditional parallel computing on a small number of powerful processors. One approach is to parallelize not the neural network itself, but the process of its training, which is the most numerically intensive part in neural network computing. During the batch training each input pattern/signal is presented to the neural network, a response is obtained and evaluated, and a direction of network parameters change (the cost function gradient) is calculated using the backpropagation algorithm. The goal is obtained parallelizing MATLAB's matrix multiplication routine. The message passing interface is used for the parallel implementation. The implementation allows to parallelize generally non-parallel procedures offered by MATLAB.