Optimization of Spark Data Skew in Big Data Environment
Huai-Chih Wang, Huali Wang · 2023
In order to solve the problem of uneven data distribution in the shuffle stage of Spark distributed platform, this paper first analyzes the causes of data skew in Spark platform, and then establishes a skew model that can quantify the degree of data skew. Based on quantifiable skew model, a shuffle partition scheme is proposed to optimize data skew in Spark distributed platform. The working node of the partitioning scheme first samples the output data of the Map stage, summarizes the data of each working node to the master node to predict the data size of all working nodes, and then pre-partitions the intermediate data according to the load balancing partitioning algorithm and Hash partitioning algorithm to obtain a pre-partitioning table. The master node distributes the pre-partitioning table to each working node. Finally, the working node partitions all the intermediate data according to the pre-partitioning table. The experimental results under different skew conditions show that the shuffle partition scheme proposed in this paper is universal and efficient, and can effectively deal with the data skew problem of Spark distributed platform.