Optimizing, Planning and Executing Analytics Workflows over Multiple Engines
Katerina Doka, Maxim Filatov, Victor Giannakouris, Verena Kantere, Nectarios Koziris, Christos Mantas, Nikolaos Papailiou, Vassilis Papaioannou, Dimitrios Tsoumakos · 2016
Big data analytics have become a necessity to businesses worldwide. The complexity of the tasks they execute is ever increasing due to the surge in data and task heterogene-ity. Current analytics platforms, while successful in har-nessing multiple aspects of this “data deluge”, bind their efficacy to a single data and compute model and often de-pend on proprietary systems. However, no single execution engine is suitable for all types of computation and no single data store is suitable for all types of data. To this end, we present and demonstrate a platform that designs, optimizes, plans and executes complex analytics workflows over multi-ple engines. Our system enables users to create workflows of variable detail concerning the execution semantics, depend-ing on their level of expertise and interest. The workflows are then analysed in order to determine missing execution semantics. Through the modelling of the cost and perfor-mance of the required tasks over the available platforms, the system is able to match distinct workflow parts to the ex-ecution and/or storage engine among the available ones in order to optimize with respect to a user-defined policy. 1.