Spindle: Techniques for Optimizing Atomic Multicast on RDMA
Sagar Jha, Lorenzo Rosa, Ken Birman · 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS) · 2022
Modern networking technologies such as Remote Direct Memory Access (RDMA) promise huge speedups in I/O bound platforms, but software layering overheads must first be overcome. Our paper studies this issue in a system that replicates small data objects using atomic multicast: a case in which internal synchronization is unavoidable, and any delay will be particularly impactful. Spindle, the methodology we propose, entails a series of optimizations including memory polling integrated with novel sender and receiver batching techniques, null-message send logic, and improved multi-thread synchronization. We applied Spindle to Derecho, an open-source library for atomic multicast, and obtained significant performance improvements both for the library itself and for an OMG-compliant avionics DDS layered on it. Derecho’s multicast bandwidth utilization for 10KB messages rose from 1GB/s to 9.7GB/s on a 12.5GB/s network, and it became more robust to delays even as latency dropped by nearly two orders of magnitude. While our focus is on the Derecho library and the OMG DDS, the same techniques should be relevant to databases, file systems, and IoT infrastructures.