An Energy-Efficient and Heterogeneous Environment Adaptive Data Layout Strategy for MapReduce
Liao Bi · Acta Scientiarum Naturalium Universitatis Sunyatseni · 2015
The problem of high energy consumption producing from big data processing is an important issue that needs to be solved,especially under the background of data explosion. Based on analyzing problems of the existing data layout policy,the problems of the in adaptation of energy-saving mode based on storage area division and heterogeneous HDFS cluster,the inflexibility of data block segmentation algorithm,the randomness of storage node selection,proposing a data layout strategy orienting to energy conservation are analyzed. Firstly,the new strategy divides the cluster into two different storage areas to meet the needs of saving energy: Active-Zone and Sleep-Zone; secondly,the new strategy has made im-provements on traditional data block computing method,proposes a minimum number of jobs calculation method to determine the number of data blocks; at last,the new strategy can increase the adaptability of the heterogeneous cluster environment and can choose the appropriate storage nodes according to different job types. Experimental results show that the new data layout strategy can adapt to the heterogeneous cluster environment and reach the goal of reducing energy consumption for MapReduce jobs.