Simulation of the Performance of MapReduce Jobs
Tiago André da Rocha Ferreira · Open Repository of the University of Porto (University of Porto) · 2016
The MapReduce distributed computation paradigm introduced by Google made a huge frenzy and with the arrival of an open source solution, Apache Hadoop framework, major companies rapidly joined this solution. Today companies such as LinkedIn and Facebook use Hadoop has the main data processing tool. In the State of the Art was found that there is a massive research in MapReduce, and that there is a lot o research to be done, and innumerous challenges ahead. This dissertation presents a solution to simplify the research and testing of algorithms for the MapReduce framework. The solution proposed is a tool for the simulation of MapReduce jobs in a cluster. The solution is based on the Apache Hadoop framework, and is highly customizable and allows the user to specify his how algorithms. The simulation environment is built on top of the grid simulator SimGrid. The SimGrid simulator is a large scale distributed systems simulator for peer-to-peer and cloud systems, that incorporates a network representation scheme. This network representation allows for a network analysis in the MapReduce jobs.