A Smart Strategy for Speculative Execution Based on Hardware Resource in a Heterogeneous Distributed Environment
Qi Liu, Weidong Cai, Zhangjie Fu, Jian Shen, Nigel Linge · International Journal of Grid and Distributed Computing · 2016
MapReduce, as a popular programming model for processing large data sets, has been widely applied.MapReduce 2.0 (MRV2) is a newly adopted one, which has a better performance.Those machines which have a lower performance in a cluster usually play a role who pull down the pace of job execution time.Speculative execution known as an approach for dealing with the above problems works by backing up those tasks running on a low performance machine to a higher one.Although multiple speculative execution strategies have been proposed, there are still a lot of pitfalls existing in the strategies.In this paper, Some pitfalls in proposed strategy have been modified and computer hardware has been taken into consideration (HWC-Speculation).In Hadoop-2.6,we have implemented it, called Hadoop-HWC.Experiment results show that our method can find a slow task correctly, also, the performance of MRV2 is improved.