Flink Task Scheduling Based on LBGA
Qingzhuo Cao, Ping She, Xiaoli Chai · 2023
In order to solve the problem of uneven task load of different nodes affecting task execution time and throughput caused by the Flink's default scheduling strategy, a task scheduling strategy based on load balancing genetic algorithm (LBGA) in the Flink streaming computing environment is proposed. The strategy starts by calculating resource node performance metrics based on the resource data monitored by the Ganglia module. Considering the matching relationship between task resource requirements and performance metrics, this paper proposes an idea of applying the genetic algorithm (GA) to a streaming computing platform for optimisation. In addition, the LBGA scheduling is proposed on this basis and then experiments are conducted to verify the effectiveness of the algorithm. The experimental results show that compared to the default scheduling algorithm of Flink, the proposed algorithm reduces the execution time of WordCount benchmark tasks by 7.6% and 7.7% for parallelism degrees of 8 and 16 respectively. Not only is the performance of resource node load balancing optimised, but throughput is also significantly improved.