A two-component mechanism to deal with the straggler problem in Hadoop
Worachate Apichanukul · NAIST Digital Library (Nara Institute of Science and Technology) · 2017
Apache Hadoop is a distributed computing platform for processing large data sets on computer clusters.Similarly to other distributed platforms, Hadoop confronts with a well-known problem called the issue of stragglers in which some tasks of a job take unusual long execution time and delay the whole job.The slow tasks are called straggler tasks.In this dissertation, we propose a two-component mechanism to deal with the problem.The first one is called a detection mechanism.This mechanism is developed based on a speculative algorithm, which is provided by Hadoop.However, the default speculative algorithm achieves low performance to classify slow tasks.We therefore propose an improved version of speculative algorithm, called Accuracy Improvement for Backup Task (AIBT), to accurately identify straggler tasks.Although the problem is better treated by AIBT, it still exists.It is found through experiments that an inefficient task distribution in existing scheduling algorithms is a cause of the problem.Existing scheduling algorithms neglect the utilization level of each node.As a result, some nodes are over utilized and suffer from the straggler problem.We propose the second mechanism, called a prevention mechanism, to protect Hadoop from the problem.The prevention mechanism effectively distributes tasks to be executed on suitable nodes.To extract the suitable nodes, we use a cost function which takes both utilization level of each node and location of processed data into account.We evaluate both mechanisms by performing experiments in an actual environment.Numerical results show that our prevention mechanism works well with our AIBT detection algorithm.Both mechanisms not only relieve the straggler problem, but also reduce task execution time and increase the amount of local execution compared with the existing algorithms.