Data locality and load balancing for parallel neural network learning
Lutz Prechelt · 1995
Compilers for neural network learning algorithms can achieve near-optimal co-locality of data and processes and near-optimal balancing of load over processors for irregular problems. This is impossible for general programs, but restricting programs to that particular problem domain allows for the exploitation of domain-specific properties: The operations performed by neural algorithms are broadcasts, reductions, and object-local operations only; the load distribution is regular with respect to the (perhaps irregular) network topology; changes of network topology occur only from time to time. Compilation techniques and a compiler implementation for the MasPar MP-1 is described and quantitative results for the effects of various optimizations used in the compiler are given. Experiments with weight pruning algorithms yielded speedups of 28% due to load balancing, and of 195% due to data locality. Two other optimizations, connection allocation and selecting the number of replicates, speed ...