Elastic MapReduce Execution
Wei Xiang Goh, Kian‐Lee Tan · 2014
With increasingly larger deployments, the MapReduce framework begins to face technical deficiencies in its execution architecture. In order to cope with the management of such limits-pushing amount of resources, there are independent developments of supplementary frameworks (e.g., YARN) that isolate resource management from the job coordinations. These resource managers also expose potential increased elasticity in job execution that has not been fully exploited by the current state-of-the-art architecture. In this paper, we present an enhanced architecture for MapReduce job execution called Elastic MapReduce Execution (EMRE) that leverages on a structured peer-to-peer overlay (i.e., BATON) to induce elasticity into the job execution without compromising on fault tolerance. The execution architecture requires no modification to the original MapReduce job definition, and our experiments indicate that EMRE will greatly improve the performance of MapReduce under various execution conditions.