Resource and Job Execution Context-Aware Hadoop Configuration Tuning
Xinhe Wang, Jianlin Zhang, Yuliang Shi · 2020
MapReduce is a programming model, which is widely-used in parallel processing of big data. Its' performance is significantly affected by configuration parameters. However, the huge parameter space and interact of parameters make it impossible to explore all the parameter combinations manually. In this paper, we propose RJHCT, a novel approach to automatically tune the configuration parameters for MapReduce applications. We use random forest regression to predict the execution time of single task, and then we predict the job execution time by packing algorithm. Leveraging the prediction model, we use genetic algorithm to search the optimal configuration parameters automatically for a given MapReduce application. Experimental results demonstrated that RJHCT improves the performance of MapReduce applications by factors of 0.28x compared with the default configuration.