Towards Elastic Data Warehousing by Decoupling Data Management and Computation
Zhi Liu, Haowen Wu, Tongxin Bai, Yang Wang, Chengzhong Xu · 2020
Moving data warehouses to the cloud is what today's companies consider a trend towards cost-effective data management. To fully achieve the economic goal, the cloud data warehouse system is supposed to be able to adjust its resource provisioning to adapt to changing workload requirements. However, traditional data warehousing architecture is not flexible enough to allow on demand resource control, which severely restrains the cloud provider as well as the users to optimize total cost and maintain desired quality of service. To build the data warehouses for the cloud, new architectures should be studied. In this paper we explore such an architecture that decouples data management and data computation to facilitate the on demand resource control. By separating the two parts, the resulting system gains more elasticity and more adaptivity. For the poof of concept, we build a prototype system, called DuoSQL, based on PostgreSQL and Spark. We validate the system using the TPC-H benchmark. The experiment results show the decoupling approach is not only flexible but also has great performance potentials.