Firehose: An Algorithm for Distributed Page Registration on Clusters of SMPs
C.R. Bell, Rajesh Nishtala · 2004
This paper proposes to improve a memory registration strategy for Remote DMA operations over pinningbased networks in the context of Clusters of Multiprocessors (CLUMPS). Although existing approaches focus primarily on bandwidth as a metric for evaluating the cost of DMA page registration (or pinning), there are many levels of host synchronization that hide the true cost of registration and prevent efficient use of one-sided RDMA communications and seriously impacts the cost for small messages. Furthermore, existing approaches do not tailor their implementations for Multiprocessors that share a single network endpoint, and as such ignore to exploit the spatial and temporal locality in the remote memory pages referenced by running many SPMD parallel programs over a single network endpoint. The proposed solution extends the idea of the Firehose algorithm and proposes a new Firehose API and library for both uniprocessor and multiprocessor variants. The SMP-aware version of Firehose pursues the same design goals of minimizing host-level synchronization – or more specifically, allowing operations to complete one-sided in the common case and reverting to rendezvouslike synchronization between endpoints when operations in the uncommon case. We motivate the need for special attention for CLUMPS and supplement a complete Firehose-SMP implementation with an evaluation of its design and some of the initial performance results are promising.