Distributed Aritificial Neural Network Architectures
David A. Calvert, Jiawen Guan · 2005
The computational cost of training Artificial Neural Network (ANN) algorithms limits the use of large systems capable of processing complex problems. Implementing ANNs on a parallel or distributed platform to improve performance is therefore desirable. This work illustrates a method to predict and evaluate the performance of distributed ANN algorithms by analyzing the performance of the comparatively simple mathematical operations which are used to construct the ANN. The ANN algorithms are divided into simple components: matrix and vector multiplication, matrix processed through a function, competition in a matrix. These basic operational parts are ex- amined individually and it is demonstrated that the computation processes of distributed Neural Networks can be derived from the composition of these basic operations. Three popular network architectures are examined: Multi-Layer Perceptrons with Back- Propagation learning, Self-Organizing Map, and Radial Basis Functions network. I. INTRODUCTION The field of Neural Networks can in principle benefit from distributed computation. Applications in this area often need huge repetitive calculations and large amounts of data to generate valid results. In this work, distributed processing is used to accelerate the training of existing networks and can make the usage of larger networks viable. An analysis of several typical Neural Network architectures reveals the common computational components for these al- gorithms. Experiments are performed using distributed imple- mentations which measure the performance for each compo- nent individually and for complete ANNs constructed using these components. From the results for the components, the behavior of an ANN implemented on a distributed platform are predicted. The total run time for the ANNs training and recall operations and the performance improvement based on the number of processors are examined. Three Neural Net- works are examined in this work. These are the Multi-Layer Perceptrons with the Back-Propagation algorithm (BP), the Self-Organizing Map (SOM), and the Radial Basis Function networks (RBF). Algorithms which are most frequently found in ANNs generally rely on efficient add and multiply operations, calcu- lations of an activation function, and comparison operations. All of the Neural Networks examined in this work can have their processing described using the following operations: • Matrix multiplication, used in RBF weights calculation and testing, and in BP weight calculation.