Analysis of Task Scheduling in Hadoop MapReduce Framework
Kamalakant Laxman Bawankule, Anil Kumar Singh, Rupesh Kumar Dewaang · 2019
There are many open source platform availablefor storage and computation of big data, hadoop is one ofthem. Hadoop can be used for implementation of programmingmodel like MapReduce which is very efficient for processingthe shorter jobs with low response time. MapReduceframework, which is popular for computation of big datain parallel, distributed across the cluster. In our experimentwe are analyzing scheduling of each task in MapReduce[11]framework with the help of two applications Word count andgrep on FIFO (first in first out) scheduler, Fair scheduler andcapacity scheduler. The jobs are submitted simultaneously forexecution to analyze the task scheduling. We tried to variedthe workload as well as Map Tasks on each slaves to observethe effect on tasks scheduling. Experiment has been carriedout on text files of 1GB, 2GB and 5GB with variations inMap Tasks as 1, 2, 3, 4 and 5 on each slave nodes, beforeexecuting the jobs. In the execution, first we submit Grepfollowed by Word Count in all the above cases for differentworkloads with different Map Tasks. We observed that inFIFO scheduler jobs are submitted as per the policy firstin first out,the jobs that are submitted first will be executedfirst. In Fair scheduler and Capacity scheduler all the jobsare given the equal share of resources, means both the jobexecutes simultaneously. Observation of results can help us toconclude that in all the three schedulers i.e FIFO schedeuler,Fair schedeuler, and Capacity schedulers, FIFO schedulertakes more turnaround time for bigger data size where as itoutperforms for the shorter jobs. However, now a days muchmore big data applications are developed with MapReducemodel which requires low turnaround time for the larger jobsas well as for shorter jobs. As a result, it becomes necessaryto verify the performance of MapReduce, especially for largerjobs which is more popular now a days and which hasattracted more and more attentions from research, industryand academia.