In Big Data Era: Analysis of Hadoop Cluster Performance
Wedad M. Alawad, Awatef Salem Balobaid · 2021
In recent decades, the dependence on digital tools to assist our daily lives has been growing in almost all sectors and fields. A huge amount of data has been generated and as a result, a new era has emerged: the era of big data. To benefit from this data, several big data analysis tools have emerged, Hadoop being one of them. In this paper, we analyzed Hadoop cluster performance and investigated the effect of the cluster and dataset sizes on its performance. Furthermore, the MapReduce model was used to study and justify its benefits in the big data analysis field. The results showed that the Hadoop cluster and the MapReduce model are powerful in processing and analyzing big data. More benefits can be achieved from the analyzing time perspective when larger data sizes are used. Interestingly, we found that increasing the cluster size beyond a certain point resulted in communication issues, thereby yielding an increase in the CPU time.