IDP: An Innovative Data Placement Algorithm for Hadoop Systems
Chia‐Wei Lee, Huang Horng-Chyau, Sun‐Yuan Hsieh · Frontiers in artificial intelligence and applications · 2015
In this paper, we propose a data placement strategy to deal with the imbalanced workload problem on DataNodes. Basing on computing capability of each node in a heterogeneous Hadoop cluster, the proposed strategy can balance the data that was stored in the DataNode such that the cost of data transfer time can be tremendously reduced. As a result, the Hadoop overall performance can be greatly improved. Experimental results demonstrate that the proposed data placement strategy can highly decrease the execution time and thus improves Hadoop performance in a heterogeneous cluster.