Optimizing Distributed In-memory Storage Systems: Fault-tolerance, Performance, Energy Efficiency
M. Taleb · HAL (Le Centre pour la Communication Scientifique Directe) · 2018
Emerging technologies such as connected devices and social networking applications are shaping the way we live, work, and interact with each other. These technologies generate increasingly high volumes of data. Dealing with large volumes of data has been an important focus in the last decade, however, today the challenge has shifted from data volume to velocity: How to store, process, and extract value from data generated With the growing capacity of DRAM, service providers largely rely on DRAM-based storage systems to serve their workloads. Because DRAM is volatile, usually, distributed in-memory storage systems rely on expensive durability mechanisms to persist data.This creates trade-offs between performance, durability and efficiency in in-memory storage systems We first study these trade-offs by means of experimental study. We extract the main factors that impact performance and efficiency in in-memory storage systems. Then, we design and implement a new RDMA-based replication mechanism that greatly improves replication efficiency in in-memory storage systems. Finally, we leverage our techniques and apply them to stream storage systems. We design and implement high-performance replication mechanisms for stream storage, while guaranteeing linearizability and durability.