Imbalanced Word Counting Using MapReduce in Serverless Platform
Xiaoyue Chen, Xinyue Li, Qingyuan Zhou · 2023
MapReduce is one of the most widely used programming models for analyzing large-scale datasets. In recent years, serverless computing, especially Function-as-a-Service (FaaS), has received a lot of attention due to its advantages of scalability, low cost, and compatibility. In the serverless environment provided by the AliCloud platform, this paper investigates the different performances of MapReduce when processing massive files under different resource configurations, and proposes a solution to the data skewing problem of MapReduce. The results show that given the same other configurations, CPU cores and WORKERS have a significant impact on function response time: the larger the CPU core in the function configuration, the shorter the function response time, and the high concurrency of WORKERS can also significantly reduce the function response time. In addition, the size of the memory configuration has a negligible effect on the response time of the function. In addition, an optimization scheme is further proposed for the common data skew problem in MapReduce and explore its effect. This scheme can greatly improve the overall task execution efficiency when the data is seriously unevenly distributed.