Task Scheduling Algorithm for MapReduce Based on DAG
Tang Yi-ta · 2014
Hadoop has been the basic platform of cloud computing research,and MapReduce is the computing mode for distributed processing of big data.For heterogeneous cluster,considering MapReduce's defects in data distribution,data locality and process of the job execution,we proposed a DAG based MapReduce scheduling algorithm.The algorithm groups the nodes based on their computing ability,transforms MapReduce job execution to DAG model and improves upward ranking to achieve better accuracy and a more reasonable sequencing of task priority.By combining the computing ability of nodes,data locality and cluster utilization,choosing the proper data nodes for task distribution and execution,our algorithm shortens task completion time.The experimental result shows that the proposed algorithm can distribute data reasonably,improve data locality effectively,reduce communication overhead,shorten schedule length of set of job,thus improving utilization of cluster.