HCEm model and a comparative workload analysis of Hadoop cluster
José Benedito De Souza Brito, Aletéia Araújo · International Journal of Big Data Intelligence · 2016
This paper describes the HCEm model, designed to estimate the size of a cluster running Hadoop, in a given timeframe on cloud environments. The HCEm consists of a light optimisation layer for MapReduce jobs and a model to estimate the size of a Hadoop cluster. Additionally, this paper presents a comparative study of HCEm using similar applications and workloads in two production Hadoop clusters, the Amazon Elastic MapReduce and a private cloud in a large financial company, in order to evaluate the performance of the model in real and intensive processing environments. The estimates generated by the HCEm model and processing performed are representative and consistent, which can help researchers and engineers understand the workload characteristics of Hadoop clusters in their production environments. The performance differences shown between the real environments, confirmed that the increased sharing of physical computing host resources reduces the accuracy of the model.