Phase based Resource Aware Scheduler with Job Profiling for MapReduce
Sudhakaran Dhanya, Shini Renjith · Figshare · 2016
MapReduce has become a popular model for data mining computation in these days. MapReduce break down each job into small map tasks and reduce tasks. These tasks are executed in parallel across a large number of machines. This can significantly reduce the running time of data-intensive jobs. The existing schedulers for MapReduce are focusing on scheduling at the task-level and they offer only sub-optimal job performance. This is because tasks can have highly varying resource requirements during their lifetime. These varying requirements make it difficult for task-level schedulers to effectively utilize available resources to reduce job execution time. To address this limitation, a fine-grained, phase and resource-aware MapReduce Scheduler was introduced that divides tasks into phases, where each phase has a constant resource usage profile, and performs scheduling at the phase level. Job profiling is not considered in this scheduler. To meet the need for job profiling, Starfish, a job profiler can be built into the system. Starfish can provide accurate resource information that can be used by the scheduler so that it can take effective scheduling decisions and reduce the job execution time. By using such a job profiler integrated within the scheduler, the performance of the scheduler is effective than the existing task-level schedulers.