Auto-Tuning Flink Configurations Based on GBDT
Xiaojun Sun, Jian Zhang, Jiaqi Wang · 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC) · 2022
Apache Flink is widely used in distributed computing scenarios, and its real-time performance is more powerful than Spark, so to maximize the performance of Flink, we need to set the relevant configuration parameters reasonably. However, it is quite difficult to select and tune the appropriate parameters from many parameters. To improve the overall efficiency of computation, it is important to study the performance modeling and Auto-Tuning Flink Configurations. In this paper, we analyze the key parameters affecting the execution time of Flink tasks and build a performance model to automatically optimize the Flink parameters based on the performance modeling. By extracting the parameters that have an impact on performance during job execution, we use GBDT to build a performance prediction model for a specific task, and finally use Particle swarm optimization (PSO) to find the optimal combination of parameters based on the performance model. Experimental results show that the tuned execution time is reduced by an average of 30% compared to the default configuration parameters.