Comparative Evaluation of Fine- and Coarse-Grain Software Distributed Shared Memory
Sandhya Dwarkadas, Kourosh Gharachorloo, Leonidas I. Kontothanassis, Scott R. Stets · 1998
With the advent of commercial symmetric multiprocessors (SMPs) and low-latency, high-bandwidth networks, clusters of SMPs provide a natural base for software distributed shared memory (S-DSM). Two distinct approaches can be used: instrumentation-based approaches that provide fine-grain coherence, and VM-based approaches that restrict the underlying coherence unit to a page, but avoid instrumentation overhead. In this paper, we compare and evaluate these approaches using two mature S-DSM systems, Shasta and Cashmere, both of which run on a DEC AlphaServer SMP cluster connected by a very low-latency, high-bandwidth, remote-write, Memory Channel network. Our results, obtained on a 16-processor, 4-node cluster, clearly illustrate the tradeoffs between fine-grain and VM-based S-DSM. A fine-grain system such as Shasta offers an easy migration path for programs developed on a hardware DSM (H-DSM). It supports H-DSM memory models, and is better able to tolerate fine-grain synchronization. In contrast, for the same unmodified programs, Cashmere’s performance is highly sensitive to the presence of fine-grain synchronization, while providing a performance edge for applications with coarse-grain synchronization. With program modifications that take coherence granularity into account, the performance gap between the two systems can be bridged. Remaining performance differences are dependent on program structure: a high degree of false sharing at a granularity larger than a cache line favors Shasta, while finer-grain false sharing and large amounts of mostly private data favors Cashmere. c Digital Equipment Corporation, 1998 This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of the Cambridge Research Laboratory of Digital Equipment Corporation in Cambridge, Massachusetts; an acknowledgment of the authors and individual contributors to the work; and all applicable portions of the copyright notice. Copying, reproducing, or republishing for any other purpose shall require a license with payment of fee to the Cambridge Research Laboratory. All rights reserved. CRL Technical reports are available on the CRL’s web page at http://www.crl.research.digital.com. Digital Equipment Corporation Cambridge Research Laboratory One Kendall Square, Building 700 Cambridge, Massachusetts 02139