Cost-Efficient Stream Processing on the Cloud
Tri Minh Truong, Aaron Harwood, Richard Sinnott, Shiping Chen · 2019
Under dynamic workload, cost-efficient stream processing on the Cloud, where the cost for Cloud resources is minimized and the performance is optimized, is non-trivial to achieve. To tackle this, we make use of queueing theory to model the performance of stream processing components. We propose a stable performance and cost trade-off scaling algorithm for large-scale stream processing on the Cloud. Our approach takes into consideration typical characteristics of Cloud environments such as the machine size, dynamic cost, high machine start-up costs and large migration time of Cloud resources. We present a strategy based on a multi-step configuration to enable dynamic stream processing on the Cloud while cost-efficiently achieving user-level Quality of Service (QoS). Our proposed solution addresses the issues of resource provisioning with respect to both the performance objectives and dynamic scheduling of Cloud environments. We present an exemplar implementation, based on a Heron stream processing application that was deployed automatically and subsequently monitored to ensure its QoS.