The design and analysis of DASH: a scalable directory-based multiprocessor
Daniel E. Lenoski · 1992
Scalable shared-memory multiprocessors combine the ease of programming of a single address space with the ability to scale performance as the system grows from a few processors to a few thousand processors. The major challenge in such machines lies in the design of a memory system that has both low latency and high bandwidth. Directory-based cache coherence can potentially satisfy both of these goals. Caching provides low latency, while the directory supports coherence without compromising bandwidth. The focus of this dissertation is the design of distributed directory-based multiprocessors. The vehicle used to drive the investigation has been the implementation of the DASH prototype system. The detailed design of the prototype has shown that while directory-based systems are complex, they are feasible. The issues that must be addressed in their design include: providing scalable memory bandwidth with limited overhead cost, reducing and hiding the latency of a physically large memory system, and managing the concurrency problems introduced by the highly parallel execution environment. The DASH prototype is the first operational machine to include a scalable cache-coherence mechanism. Analysis of the prototype has led to new insights into directory-based systems. First, the prototype shows that the complexity of directory-based protocols is manageable and that the hardware overhead can be kept quite low (less than 20% in the prototype). Second, while a single-address space can simplify use of a parallel system, it does not imply that interprocessor communication is free. Remote memory latency is significant ($>$100 processor clocks in the prototype), and good cache and memory locality are important to achieve high performance. Third, and most important of all, running parallel applications on the prototype has demonstrated that high-performance and near linear speedups are possible on this class of machine. Finally, the preliminary design of a very large-scale highly integrated DASH system indicates that the architecture can be scaled in today's technology to a system with over 2000 processors and 1 TeraOPS in peak performance.