Analysis, Modeling, and Simulation of Hadoop YARN MapReduce
Thomas C. Bressoud, Qiuyi Tang · 2016
Despite much work in simulation of MapReduce applications and in optimizing scheduling to minimize workload execution time, we still need models and tools to understand the performance of the individual tasks of a MapReduce application. Armed with this knowledge, we can characterize applications and model task operation to determine their performance in a given cluster setting. We have developed and analyzed a model of MapReduce tasks that abstracts their operation at a balanced level, avoiding too much detail that slows down analysis and requires knowledge of the inner workings of the cluster system, but at sufficient detail to be able to generalize to multiple application classes and cluster systems. The model, and an accompanying simulator, are evaluated against execution of real MapReduce applications, and show very promising results.