Analysis of MapReduce operation in Hadoop YARN and Rack-Aware Resource Management System for YARN
T. Moses, T. A. Badmos, R. Abdulkarim · International Journal of Darshan Institute on Engineering Research and Emerging Technologies · 2022
Hadoop MapReduce has been the major computational paradigm for the analysis and exploration of big data.It facilitates concurrent processing of big data by splitting data into chunks and processing these data using commodity cluster processors.The processed data are aggregated from the multiple commodity clusters to return a consolidated output.Hadoop YARN has been a major enabler for this computation but its central resource manager is a bottleneck.Rack-aware resource management system developed to overcome this bottleneck decentralized the responsibilities of the resource manager by providing another layer in the architecture of Hadoop called Rack Unit Resource Manager Layer.This work, therefore, analyses the MapReduce operation between these two architectures to understand the behaviour of each data chunk (block).Wordcount operation was used for this analysis and the result obtained showed that the rack-aware system performed better as data grows bigger.