Automatic optimisation of MapReduce designs by geometric programming
Qiang Liu, Tim Todman, Wayne W. Luk, George Anthony Constantinides · 2009
Many important applications can be expressed using the MapReduce pattern, where a computation is decomposed into a map phase on which each element of source data is independently operated, followed by a reduce phase in which the mapped elements are combined with an associative operator. We develop an approach for compiling applications with the MapReduce pattern into parallel hardware. Using optimisation techniques based on geometric programming, we map the computation onto a resource-constrained architecture. Furthermore, we explore important variations of MapReduce, such as making the reduce a linear structure rather than a tree structure. Results for four benchmarks show that our approach can improve system performance by up to 170 times compared to the initial designs.