A Reduce Task Scheduler for MapReduce with Minimum Transmission Cost Based on Sampling Evaluation
Xia Tang, Lijun Wang, Zhiqiang Geng · International Journal of Database Theory and Application · 2015
MapReduce is a popular framework for processing large datasets in parallel over a cluster.It has gained wide attention for its high scalability, reliability and low cost.However, its performance may be degraded by excessive network traffic when processing jobs, for such two problems as data locality in reduce task scheduling and partitioning skew.We propose a Minimum Transmission Cost Reduce task Scheduler (MTCRS) based on sampling evaluation to solve the two problems.The MTCRS takes the waiting time of each reduce task and the transmission cost set as indicators to decide appropriate launching locations for Reduce tasks.The transmission cost set is computed by a mathematical model, in which the parameters are the sizes and the locations of intermediate data partitions generated by Average Reservoir Sampling (ARS) algorithm.The experiments show that the MTCRS reduces network traffic by 8.4% compared with Fair scheduler.