MAP task allocation strategy in an ARM-based Hadoop cluster by using local storage as split cache
Bongen Gu, Yoonsik Kwak · International Journal of Advanced Media and Communication · 2016
The increase of power consumption makes the cost of cluster operation higher.One approach for reducing power consumption is to establish a cluster with small nodes which equip a low-power, high-performance processor.Since many lowpower consumed nodes do not have storage devices, a separate storage system is required to store large-volume data while nodes mount this storage space to save data.When a Hadoop cluster is configured in such a condition, each node's access to a storage results in excessive network load and delays the execution of Hadoop Map tasks.In this study, we propose a newmap task scheduling policy for Hadoop.This policy transmits multiple splits to nodes at once to reduce network load.In addition, local storage space of nodes is used as a cache for a split, which shortens the time to access splits, so this policy can reduce the execu tion time of Hadoop applications.