Performance-sensitive Data Distribution Method for Distributed Stream Processing Systems
Yanqiu Chen, Linjiang Zheng, Weining Liu · 2020
In the distributed stream processing, data skew and dynamics can result in imbalanced load distribution at downstream tasks and affect the throughput of the systems. Efficient data distribution method is urgently needed to solve the problem of imbalanced load distribution to improve system throughput. Existing researches cannot better balance the semantic correctness and the throughput of distributed stream processing systems, and it is hard to apply to distributed clusters with different node performance. In this paper, we propose a performance-sensitive data distribution method. Firstly, we propose a load balancing framework that considers the performance of nodes. Then we present a data redistribution algorithm, which aims to balance the execution delay between parallel nodes and follows the principle of "Less Migration key, Less Partition key"(LMLP). The proposed algorithm can reduce the extra cost caused by the data redistribution process and it can be applied to distributed clusters with different performance. Experimental results show that our method is better than the similar methods in terms of load imbalance degree, complete latency, routing table size, etc. And we also use real data to prove that our method can obtain higher throughput and resource utilization.