Dynamic Range Partitioning with Asynchronous Data Balancing
Djahida Belayadi, Walid Khaled Hidouci · 2016
Our last decade has experienced significant growth in terms of data generated by billions of people connected to the Internet. Recent prognoses about Big Data, Internet of Thing, Cloud Computing show growing demand for an efficient processing of huge amount of data with strict time limits. The Best data distribution on Shared Nothing Architecture (SN) is a major issue. SDDS (Scalable Distributed Data Structure) is one of the most important data structure family that is able to store, manage a huge amount of data in distributed environments. In this paper, we present an SDDS driven approach in which data is partitioned using the dynamic intervals. We are targeting any application that handles a large volume of data to be distributed over a cluster of parallel processing. This volume of data is continuously supplied in real time from multiple sources. In such situation, to face the risk of major data imbalances (Data Skew) that would influence negatively the performance of parallel processing (such as range queries), we propose a self-balancing range partitioning scheme allowing nodes processing to dynamically adjust the boundaries of their ranges. Experimental results show that our approach significantly improves processing time, specially for data sorting operation, keep data balanced over the cluster.