MRAbF: MapReduce Resource Allocation Optimization Algorithm Based on Fair Policy
Yaping Wan, Zhihui Peng, Huajuan Chen, Wangda Yang · 2023
In the era of big data, data storage and computation have become a mainstream issue. Hadooop, as a distributed computing framework capable of handling large-scale datasets, the computing performance of its computing component MapReduce greatly affects the efficiency of big data processing. However, the current MapReduce computing component has the problem of uneven resource allocation during the Map computing phase. By analyzing the calculation process of MapReduce in the article, it is concluded that two main resource allocation parameters, mapreduce. task. io. sport. mb and mapreduce. map. sport. spool. percentage, affect the computational performance of the Map phase. Thus, a MapReduce resource allocation optimization algorithm MRAbF based on fairness strategy was proposed. By comparing the WordCount calculation experiment with the native MapReduce, the optimized MapReduce calculation performance can be improved by 4.8% to 17.2%.