Load Balancing Strategy on MapReduce with Locality-aware
LI Hang-che · 2015
Intermediate data distribution characteristics and network traffic overhead are not considered in any existing research on load balancing strategy on MapReduce,resulting in additional network traffic overhead and decrease of system efficiency.To solve this problem,this paper presented a locality-aware load balancing strategy.By taking advantage of the new features of resource management brought by YARN,the strategy can obtain the data distribution when the buffered data are written to local disk.The strategy schedules the reduce tasks according to the data distribution along with the processing speed of each node to decrease network overhead while maximizing load balancing of each node.In addition,to further improve the performance of scheduling strategy with data skew,this paper introduced the strategy of fine-grained partitioning and self-adaption fragmentation.The comparative experimental results show that the presented strategy can improve the performance effectively,and reduce the total network traffic overhead.