Adaptive Fault-Tolerance for Dynamic Resource Provisioning in Distributed Stream Processing Systems
Paolo Bellavista, Antonio Corradi, Spyros Kotoulas, Andrea Reale · 2014
A growing number of applications require continuous pro-cessing of high-throughput data streams, e.g., financial anal-ysis, network traffic monitoring, or Big Data analytics for smart cities. Stream processing applications typically re-quire specific quality-of-service levels to achieve their goals; yet, due to the high time-variability of stream characteris-tics, it is often inefficient to statically allocate the resources needed to guarantee application Service Level Agreements (SLAs). In this paper, we present LAAR, a novel method for adaptive replication that trades fault tolerance for in-creased capacity during load spikes. We have implemented and validated LAAR as a middleware layer on top of IBM In-foSphere Streamsr. We have performed a wide set of exper-iments on an industrial-quality 60-core cluster deployment and we show that, under the assumption of only statistical knowledge of streams load distribution, LAAR can reduce resource consumption while guaranteeing an upper-bound on information loss in case of failures.