Synchronization and load imbalance effects in distributed memory multi‐processor systems
Dan Cristian Marinescu, John R. Rice · Concurrency Practice and Experience · 1991
Abstract Synchronization is a major cause of wasted computing cycles and of diminished performance in parallel computing. This paper investigates the effects of synchronization upon the performance of iterative methods on distributed memory MIMD machines. A quantitative analysis of the effects of the communication latency and of the load imbalance due to the non‐deterministic execution times for iterative methods is presented. This analysis explains the rather poor performance observed often in actual implementations of such methods and suggests better ways to achieve convergence without frequent synchronization.