Scheduling for response time in Hadoop MapReduce
Xiangming Dai, Brahim Bensaou · 2016
In this paper, we propose a novel task scheduling algorithm for Hadoop MapReduce called dynamic priority multiqueue scheduler (DPMQS). DPMQS i) increases the data locality of jobs, and, ii) dynamically increases the priority of jobs that are near to completing their Map phase, to bridge the time gap between the start of the reduce tasks and the execution of the reduce function for these jobs. We discuss the details of DPMQS and its practical implementation, then assess its performance in a small physical cluster and large-scale simulated clusters and compare it to the other schedulers available in Hadoop. Both real experiments and simulation results show that DPMQS decreases significantly the response time, and demonstrate that DPMQS is insensitive to changes in the cluster geometry.