How to program 122,400 heterogeneous cores and retain your sanity
Scott Patkin · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2010
Current technology trends favor hybrid architectures, typically with each node in a cluster containing both general-purpose and specialized 'accelerator' processors. The typical model for programming such systems is host-centric: The general-purpose processor orchestrates the computation, offloading performance-critical work to the accelerator, and data is communicated only among general-purpose processors. In this talk we propose a radically different hybrid-programming approach, which we call the 'reverse-acceleration model'. In this model the accelerators orchestrate the computation, offloading unacceleratable work to the general-purpose processors. Data is communicated among accelerators, not among general-purpose processors. We present the Cell Messaging Layer (CML), an implementation of the reverse-acceleration model for Los Alamos National Laboratory's Roadrunner supercomputer, a complex conglomerate of 122,400 processor cores of various types, multiple memory domains, and multiple network types, all with radically different performance characteristics but which together make Roadrunner the world's second-fastest supercomputer. CML demonstrates a new messaging-layer implementation technique called 'receiver-initiated message passing', which reduces communication latency by up to a third. Our thesis is that the reverse-acceleration model simplifies porting codes to heterogeneous systems and facilitates performance optimization. We present a case study of a legacy neutron-transport code that we modified to use reverse acceleration. Performance results from running this code across the full Roadrunner system indicate a substantial performance improvement over the unaccelerated version of the code.