An Efficient Job Scheduling for MapReduce Clusters
Jun Liu, Tianshu Wu, Ming‐Wei Lin, Shuyu Chen · International Journal of Future Generation Communication and Networking · 2015
The job scheduling for Map Reduce clusters has received significant attention in recent years, because it plays an important role on Map Reduce clusters.Traditional job scheduling performs poorly in assigning a task to appropriate nodes, and can not predict the resource utilization of the unexecuted tasks.To address the problems, an efficient job scheduling for Map Reduce clusters is proposed in this paper.The job scheduling introduces dynamic priority scheduling and real-time prediction model.Dynamic priority scheduling introduces the minimum cost data locality algorithm with a weight to deal with different size jobs, and real-time prediction model can predict the resource utilization of unexecuted tasks by calculating the running tasks.The resource utilization contains CPU, memory, and network.Experimental results prove that the proposed job scheduling is able to perform well in Map Reduce clusters.