Efficiency Assessment of MapReduce Algorithm on a Serverless Platform
Jingwen Cai, Kai Huang, Zhuoya Liao · 2023
The MapReduce programming approach can divide large-scale data processing tasks into smaller, independent tasks, run those tasks concurrently on server clusters, and then integrate the outputs of those tasks to get the final result. Considering the wide application of MapReduce in massive data processing, this paper applies it to the serverless platform emphasizing ideas and services for realization simulation and applies it to the word count experiment. In the experiment, MapReduce is implemented on Alibaba Cloud and validated using a word count experiment of about 11K words. Its execution time on the platform under different CPU cores, memory configurations, and the number of workers is validated. By changing different platform configurations, it is concluded that compared to the number of workers, the size of CPU core has a huge impact on the response time, and the memory configuration does not affect the execution time of the model to a certain extent. By adjusting the model parameters in the experiment, the serverless implementation has a high efficiency compared with the non-serverless MapReduce model. The relationship between implementation efficiency and platform configuration provides references for optimizing the application of the MapReduce model on the serverless platform. Moreover, it may inspire the configuration of serverless hardware support in the future. At the same time, it further reflects the serverless platform's optimization of resource utilization efficiency in machine learning training.