Efficient Processing of Job by Enhancing Hadoop Map Reduce Framework Using Containers
Shweta V. H. Chaudhari, Revati Machindra Wahul · International Journal on Communications Antenna and Propagation (IRECAP) · 2017
Energy efficiency is mainly considered as a primary reason for migrating to Cloud environment. Hadoop is not only a Big Data processing system but also a storage system. Currently, huge effort has been invested to enhance Hadoop performance by considering different factors like job scheduling, improving HDFS mechanism, data locality, job execution time, system memory, data caching. But these experiments are seen less effective to enhance the Hadoop performance. The objective of this work is to focus on the network latency of Hadoop to enhance its performance by using the docker technique. The Docker is an open stage for delivery, creating and running applications. It is able to bundle and to package and run an application in a loosely isolated environment called a container. In our study, containers are running the Hadoop framework in a private Cloud. The workload is distributed among the containers in a single system. The workload depends on distributing HDFS Data. The goal is to enhance the performance of MapReduce framework of Hadoop by considering factors like the number of reading operations, the number of writing operations and CPU processing time using the different job with different data set size. In the proposed experiment, the same job with different size of the dataset was used.