Similarity-Based Node Distance Exploring and Locality-Aware Shuffle Optimization for Hadoop MapReduce
Jihe Wang, Danghui Wang, Meng Zhang, Meikang Qiu, Bing Beverly Guo · 2017
To shorten the networking delay from MapTracker to ReduceTracker has attractive potential to gain high performance shuffle for MapReduce. As the original MapReduce shuffle has no locality-aware feature when assigning reduce-tasks over computing nodes, we plan to present a similarity-based distance in the proposed Cloud Node Space to evaluate distance between two computing nodes in data center. Then, we implement a centralized and statistic-based locating service prior networking shuffle to place reduce-tasks near their corresponding data. Experimental results show that, comparing with Hadoop version, this service can achieve 2.3X speedup on shuffle time and bandwidth budget decreases by 60%.