Study on adaptive delay schedule algorithm based on progress control of Hadoop
Junyi Lv, Pingjian Zhang · 2017
The existing Hadoop job scheduling algorithms may improve the performance of MapReduce in many ways. But they seldom guarantee good data locality which is very import to performance by minimizing the network cost. To address the problem above, in this paper, an adaptive delay-scheduling algorithm based on progress control is proposed to maximize the performance of the data locality from the point of view of optimizing delay scheduling. It can not only calculate the appropriate dynamic delay for job in various network condition, cluster state, but also control the delay of future tasks based on the job progress and delay comparison analysis, which can avoid some extreme cases happen like some job may wait overtime sometimes due to a probable forecast deviation. The results of experiment demonstrate that the algorithm is effective.