PTC: Performance Tuning Component for Auto-Tuning of MapReduce's Configuration
Ramesh Prasad · NORMA · 2017
Hadoop's MapReduce framework was developed to process large datasets in a distributed environment. Performance of MapReduce job is driven by large number of settings and configuration parameters. Manual configuration of these parameters and identification of optimal values is an error prone and tedious task. Improving Performance of MapReduce framework is important in order to effectively utilize the resource. In this work, existing research methodologies has been evaluated to understand the impact of these configuration parameters and approaches to identify their optimal values. In this research, we propose Performance Tuning Component for Auto-Tuning of configuration parameter with optimal values of io.sort.factor and mapreduce.job.reduces parameters. Prediction model has been developed in order to find the optimal values of these parameters using Ridge Regression Algorithm. Prediction model has accuracy of 91.26% for predicting values of configuration parameters. TeraSort and WordCount have been used as benchmark for evaluation of performance for MapReduce job. Experiments results show that the proposed solution provided about 6-48% improvement in execution time for different MapReduce jobs executed. Cost of MapReduce in cloud is associated with the time spent and resources utilized by MapReduce job. Hence, our proposed solution will not only save time, but also cost associated by reducing the execution time of the MapReduce job.