Implementation and Evaluation of the JobTracker Initiative Task Scheduling on Hadoop
Kazuki Yamazaki, Ryota Kawashima, Shoichi Saito, Hiroshi Matsuo · 2013
MapReduce is one of the major successful framework to process large-scale data efficiently. Distributed programs can be implemented easily by describing only two methods, Map and Reduce. In Hadoop which is an open source implementation of MapReduce, a JobTracker (master program in Hadoop) assigns Map Tasks and Reduce Tasks to TaskTrackers (slave programs which execute the tasks). In an environment that multiple Hadoops are running on a physical machine, its computational resources should be shared by every Hadoop (Multi-Hadoop environment). In this environment, available computational resources of each Hadoop fluctuate dynamically by behaviors of other Hadoops. Therefore, the JobTracker needs to decide assignment of tasks based on loads and available computation resources on the cluster (JobTracker Initiative Task Scheduler). In this paper, we propose a method which decides the number of task executions in order to use computational resources efficiently based on a load on each computer. And we evaluate its performance, and our results show that the proposal method has achieved a reduction of execution times of jobs by about 11.1% in Multi-Hadoop environment as compared to original Hadoop.