ESDB: Processing Extremely Skewed Workloads in Real-time
Jiachi Zhang, Shi Cheng, Zhihui Xue, Jianjun Deng, Cuiyun Fu, Wenchao Zhou, Sheng Wang, Changcheng Chen, Feifei Li · Proceedings of the 2022 International Conference on Management of Data · 2022
With the rapid growth of cloud computing, efficient management of multi-tenant databases has become a vital challenge for cloud service providers. It is particularly important for Alibaba, which hosts a distributed multi-tenant database supporting one of the world's largest e-commerce platforms. It serves tens of millions of sellers as tenants, and supports transactions from hundreds of millions of buyers. The inherent imbalance of shopping preferences from the buyers essentially generates a drastically skewed workload on the database, which could create unpredictable hotspots and consequently large throughput decline and latency increase. In this paper, we present the architecture and implementation of ESDB (ElasticSearch Database), a cloud-native document-oriented database which has been running on Alibaba Cloud for 5 years as the main transaction database behind Alibaba's e-commerce platform. ESDB provides strong full-text search and retrieval capability, and proposes dynamic secondary hashing as the solution for processing extremely skewed workloads. We evaluate ESDB with both simulated workloads and real-world workloads, and demonstrate that ESDB significantly enhances write throughput and reduces the completion time of writes without sacrificing query throughput.