Communication Optimization of Compute-Intensive Clusters Based on Software-Defined Networks
Haifeng Wang, Yunpeng Cao · 2019
Data transmission among nodes in compute-intensive cluster with MapReduce is a major performance bottleneck. In order to optimize network communication performance, a distributed data mapping model is constructed for big data computing jobs. Firstly, the mapping model extracts the spatio-temporal features of the input data of computing jobs, and optimizes job data layout by replacing storage locality with communication locality. Secondly, decision-space transformation is used to classify calculation jobs, and big data calculation jobs are divided into two categories: communication-intensive and non-intensive. Finally, the data communication of computing cluster is managed by software-defined network, and communication-intensive jobs are mapped to nodes of high-quality link by using global sensing capability of software-defined network to improve the communication performance of intermediate data. Experiments show that the model has better communication optimization effect for data communication-intensive jobs, and data transmission delay is reduced from 4.2% to 5.7%. Therefore, this communication optimization scheme is suitable for data center or large-scale cluster communication optimization, and adapts to various big data scheduling strategies and multiple network topologies.