A Genetic Optimization Physical Planner for Big Data Warehouses
Soumia Benkrid, Yacine Mestoui, Ladjel Bellatreche, Ordonez Carlos · 2020
Workload-driven approaches for partitioning and tuning traditional Parallel Database systems are well studied in the literature. Unfortunately, in the context of new generation "Big Data" warehouses, these approaches are not correctly adapted to Business Intelligence 2.0, where the analyst is at the heart of decision support systems. This "disconnect" situation strongly impacts both data partitioning and fragment allocation processes, which are essential to achieve good query performance. To overcome this problem, recent studies proposed online data partitioning and fragment allocation using AI techniques to improve query performance with adaptive behavior. Nevertheless, they have important limitations: they add significant overhead and they tend to focus on the current workload, ignoring query logs. With such motivation in mind, we first formulate the problem of optimizing database partitioning subject to feasibility constraints, based on a query workload. We then introduce a proactive partitioning approach combining offline and online processing phases, inspired by closed-loop control (used in engineering disciplines) and genetic algorithms (from AI). We present an experimental validation on a big data cluster that shows promising results on typical OLAP workloads.