Node-capability-aimed Data Distribution Strategy in Heterogeneous Hadoop Cluster
Lin Chang-han · Journal of Chinese Computer Systems · 2015
The current implementation of Hadoop focuses on homogeneous cluster,and assumes that most tasks can obtain data locally.However,most clusters are heterogeneous in practical. This exposes the fact that the existing data allocation strategy does not fully consider data locality,which leads to unnecessary data transmission which will occupy extra network bandwidth and cost time for transmission.By combining the relation between data placement and task execution in Hadoop,this paper carries out data distribution according to the different nodes' execution ability for different tasks. Considering the inherent computing property of heterogeneous cluster,we propose a data distribution strategy among racks based on task characteristics and node computing capacity. The allocation strategy pays more attention to data locality,and maximizes accesses to local data as much as possible. The experimental results demonstrate that,the allocation strategy can effectively reduce the execution time,and improve the timeliness; meanwhile,it can improve the data locality,reduce network data transmission and avoid congestion; finally,the allocation strategy also exhibits good stability.