Mapping artificial neural networks on massively parallel architectures
Qutaibah Marwan Malluhi · 1995
Recently, Artificial Neural Networks (ANNs) have received a great deal of researchers attention. One of the main motivations for this increased attention is the utilization of parallel computers which allowed ANN investigators to simulate and test their ideas in ways not available before. This research concentrates on the use of massively parallel machines for efficient implementation of neural networks. In this dissertation, we review the state of the art in the area of parallel ANN implementation. Then we present a class of efficient ANN implementation techniques. As typical ANN examples, the multilayer feedforward with backpropagation (FFBP) and the Hopfield ANN models are selected. Parallel algorithms to implement the recall and the training phases of the FFBP model as well as the recall phase of the Hopfield model are provided. A major advantage of our approach is high performance. Unlike almost all other techniques presented in the literature which require O(N) time, where N is the size of the largest layer, our implementation only requires O(log N) time. Moreover, it allows the pipelining of more than one input pattern which further improves the performance. A parallel structure, called the mesh-of-appendixed-trees (MAT), is proposed for efficient implementation of ANNs. The MAT is utilized for developing two fast special purpose neural computers. A recursive procedure to embed the MAT structure into the hypercube topology is developed. This procedure is used as the basis for an efficient mapping technique to map ANN computations on general purpose hypercube massively parallel systems. The MAT structure is based on the binary tree. Three structures based on a special kind of trees, called sheered trees, are also employed for mapping ANNs on hypercubes. The sheered tree approach is easier than the binary tree approach, demands less processor memory, and is very convenient for SIMD hypercubes. Moreover, it produces algorithms that can be easily simulated on several other hypercubic topologies like, the CCC, HHC, and shuffle exchange. For a MIMD environment, several improvements of the original SIMD algorithms are described. The different algorithm variations and alternatives are carefully studied and investigated.