Improved delay-scheduler algorithm in homogeneous Hadoop cluster
Xi Duan · Jisuanji yingyong yanjiu · 2013
Locality problem is caused by the physical location inconsistency between computing resource and data resource in Hadoop.Delay scheduling algorithm to solve locality problem which taking the physical location of task data to be processed as computing nodes and migrating task to the target nodes.However,it may appear with a work tasks focus on running in one computing node,resulting non-ideal parallelling effect in operation.To solver this problem,this paper proposed delay-capacity scheduler algorithm on the basis of delay scheduler algorithm,which allowed some task run on a node that did not contain its input data,so that decrease the job response time and improve the degree of job parallelization.Finally,through experimental analysis,the improved algorithm in efficiency and parallelization effect is obviously superior to the original delay scheduling algorithm.