CAPSys: Contention-aware task placement for data stream processing
Yuanli Wang, Lei Huang, Z. Maria Wang, Vasiliki Kalavri, Ibrahim Matta · 2025
In the context of streaming dataflow queries, the task placement problem aims to identify a mapping of operator tasks to physical resources in a distributed cluster. We show that task placement not only significantly affects query performance but also the convergence and accuracy of auto-scaling controllers. We propose CAPSys, an adaptive resource controller for dataflow stream processors, that considers auto-scaling and task placement in concert. CAPSys relies on Contention-Aware Placement Search (CAPS), a new placement strategy that ensures compute-intensive, I/O-intensive, and networkintensive tasks are balanced across available resources.