Research on optimization of job scheduling based on multi-queue for Hadoop platform
LI Chun-ya · Jisuanji yingyong yanjiu · 2014
This paper discussed some job scheduling algorithms for Hadoop platform,and proposed a novel scheduling optimization algorithm based on multi-queue which provided differentiated services based on the type of jobs. Firstly,it assigned the different type of jobs to the opposite node according to its current load so as to increasing the resource efficiency of nodes. Secondly,it allowed the available resource of job queue to be occupied by the other job queues,meanwhile,the occupied resources could be reclaimed by preemptive strategies. Finally,it could prevent the ping-pong effect by the logical division of the shared queue list and the non-shared queue list. The experiments on Hadoop cloud computing platform compared the proposed algorithm with the default native algorithms. Results show that it has a significant improvement in execution efficiency and stability of job scheduling.