Benchmarking and simulating the fundamental scaling behaviors of a MapReduce engine
Brenton D. Walker · 2017
We present MRSperf, a tool for running simple stochastically-controlled streams of jobs on a Spark cluster, and forkulator, a modular simulator for models of parallel processing. While it is common for networking researchers to build models and experiments on top of simple stochastically well-defined building blocks, research and development in MapReduce systems tends to be more divergent, either focusing on emulating realistic but very complex applications, or on theoretical models that may be far from accurate representations of real MapReduce system architecture. The purpose of these tools was originally to validate and guide the development of theoretical models in Network Calculus, but we believe the tools have more general utility. We use these tools to compare the popular Fork-Join model to the Non-Idling Single-Queue model, which more accurately reflects the behavior of the default task manager in Apache Spark, and propose future directions for their development.