Feasibility Study of Location-Conscious Multi-Site Erasure-Coded Ceph Storage for Disaster Recovery
Keitaro Uehara, Yih-Farn Robin Chen, Matti A. Hiltunen, Kaustubh Joshi, Richard D. Schlichting · 2018
A multi-site, distributed storage system deployment is frequently needed for disaster recovery. Traditional master-slave backup/restore solutions have their limitations and cannot meet the high data durability, system availability, and low latency required in many use cases. Software-defined storage (SDS) achieves flexibility, scalability, and cost-effectiveness through disaggregation of storage hardware from software and enables innovations in storage architectures. The open-source SDS Ceph solution can be used to build storage systems with high reliability using a disaster recovery (DR) configuration among multiple sites while maintaining cost-effectiveness. However, using multi-site erasure coding has the potential issue of longer read latency because of communication delays caused by data chunks distributed among multiple sites. This paper proposes location-conscious, multi-site erasure-coded Ceph storage that achieves both high reliability and lower read latency - even for sites that are geographically distant from each other. The investigation was conducted using a Ceph tool to confirm localities with the proposed Ceph CRUSH map, and experiments were conducted to evaluate I/O traffic to validate the proposed hypothesis. Our results show that the proposed location-conscious multi-site erasure-coded storage can be implemented in Ceph with both high reliability and low latency.