Prediction Model of Energy Consumption for MapReduce Based on Job Running History Logs
Liao Bi · 2015
The problem of high energy consumption produced in big data processing is an important issue that needs to be solved,especially under the background of data explosion.The energy consumption model is the basis for research to improve the energy efficiency of the MapReduce.Using traditional model to calculate the MapReduce job's energy consumption faces challenges.After research on the cluster structure,job task decomposition and task slot mapping mechanism,we proposed the prediction model of energy consumption for MapReduce based on job running history logs.Through the analysis of historical operating information of different jobs,we got the computing power and energy consumption characteristics of DataNode running different tasks,and then implemented the forecast of the energy consumption of the MapReduce job before its execution process.The experimental results demonstrate the feasibility of energy prediction model,and the purpose of improving the prediction accuracy of the model can be achieved by adjusting the correction factor.