Flexible Data-Aware Scheduling for Workflows over an In-memory Object Store
Francisco Rodrigo Duro, Javier Garcia‐Blas, Florin Isailă, Justin M. Wozniak, Jesús Carretero, Robert B. Ross · 2016
This paper explores novel techniques for improving the performance of many-task workflows based on the Swift scripting language. We propose novel programmer options for automated distributed data placement and task scheduling. These options trigger a data placement mechanism used for distributing intermediate workflow data over the servers of Hercules, a distributed key-value store that can be used to cache file system data. We demonstrate that these new mechanisms can significantly improve the aggregated throughput of many-task workflows with up to 86x, reduce the contention on the shared file system, exploit the data locality, and trade off locality and load balance.