RDMA-enabled Distributed Persistent Memory System
Youmin Chen · 2023
We design and implement DPMS (Distributed Persistent Memory Storage System) based on persistent memory and RDMA, which unifies the memory, file system and key-value interface in a single system. DPMS is designed upon pDSM, a unified distributed persistent memory abstraction. pDSM acts as a generic layer to connect the PMs of different storage nodes via high-speed RDMA network, and organizes them into a global shared address space. It provides fundamental functionalities, including global address management, space management, fault tolerance, and crash consistency guarantees. Applications are enabled to access pDSM with a group of flexible and easy-to-use APIs, by using either raw read/write interfaces, or the transactional ones with ACID guarantees. Based on pDSM, we implement a distributed file system and a key-value store named pDFS and pDKVS, respectively. Together, they uphold DPMS with high-performance, low-latency, and fault-tolerant data storage. We evaluate DPMS with both micro-benchmarks and real-world memory-intensive workloads. Experimental results show that DPMS is capable of delivering an aggregated bandwidth of 120 GB/s with 6 nodes. When processing memory-intensive workloads like YCSB and Graph500, DPMS improves the performance by one order of magnitude compared to existing systems, and keeps consistent high efficiency when the workload grows to multiple terabytes.