Study of an Improved Hadoop Speculative Execution Algorithm
Bao Yi Wang, Xiao Pu, Shao Min Zhang · Applied Mechanics and Materials · 2014
The problems of difference of nodes capabilities and the unevenly-distributed bandwidth of the network, widespread exist in the heterogeneous clouding environment. Together with the users randomness of submitting jobs, the problems above lead to server synchronization problems.Under the platform of Hadoop and the situations mentioned above, we come up with a method which is based on the native hadoop speculative algorithm to solve the problems. Through monitoring the load-balance in realtime, dynamically assessing the performance of the node and making the speculative tasks happened in high-performance node which meantime is the nearest node from input split, the algorithm effectively reduces the occupation of the network and accelerates executing speed. The experiment result shows that the method in the execution of which the speculative tasks has a high ratio, significantly improved the efficiency and throughput of the cluster.