Resd: High-Performance RDMA-enabled Control Plane for NVMe Shared-Disk Cluster
Chenming Chang, Jiawei Li, Yucheng Kang, Yupeng Chen, Yi Zhang · 2024
In cloud computing environments, providing high-performance and reliable storage solutions is critical. Shared disk is a typical product in this domain, allowing one cloud disk to be mounted on multiple machines, enabling rapid failover and even active-active configurations for customer applications. To ensure service availability and data durability, a control plane is typically established to manage QoS and implement I/O fencing for shared disks. However, the widespread adoption of RDMA and NVMe technologies in recent years has significantly enhanced cloud disk performance, rendering traditional network based control planes a bottleneck.We presents Resd, a novel RDMA-enabled control plane for NVMe Shared-Disk Cluster that bypasses server-side CPUs by using One-Sided RDMA operation with RDMA-friendly data structure in QoS token getting and persistent reservations(PR) lock acquisition/release paths. Resd consists of Resd Server and Resd Client. The RESD Server serves as an central node, storing QoS information and PR lock for shared disks. Resd Clients, deployed on compute nodes, provide interfaces for cloud disk clients to acquire QoS tokens and PR locks. To ensure the service level objectives(SLOs) of shared disks, Resd also includes an algorithm that dynamically adjusts the distribution of QoS tokens based on overall cluster pressure. Our evaluation shows that Resd can implement QoS and I/O fencing capabilities for existing NVMe shared disks with minimal performance loss. Compared to the BRPC-TCP based implementation, Resd reduces average I/O latency by 78.1% and improves throughput by 25%.