Aggrandizing Hadoop in terms of node Heterogeneity & Data Locality
S. Sujitha, Suresh Jaganathan · 2013
The growth of data has increased exponentially in recent years. In this context, data-center scale computer systems are built to meet the high storage and processing demands of these applications. Such systems are composed of hundreds, thousands, or even millions of commodity computers connected through a LAN housed in a data center. It has a much larger scale than a traditional computer cluster. Hadoop enables the distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, with a very high degree of fault tolerance. Strength of Hadoop is in its ability to detect and handle failures. The original Hadoop native task scheduler implicitly assumes that cluster nodes are homogeneous. This assumption is used to identify a slow task and re-execute it. However, this assumption does not hold where the cluster nodes are heterogeneous, since speculatively identifying a slow task will give rise to erroneous conclusions. In MapReduce, the sub-task is transferred to a node for execution. The input to the subtask, if not present in the node, must be transferred from another node in the network. Transferring data takes time and delays execution. In this paper, we have proposed a methodology for improving Hadoop in-terms of Heterogeneity and Data Locality. The performance of improved version can be measured using these metrics, i)execution time, ii)response time, iii)tasks submitted, iv)time related to jobs i.e. arrival, start and completion, v)completed task, vi)fairness, vii)locality and viii)mean completion time.