Workload characterization for operator-based distributed stream processing applications
Xiaolan J. Zhang, Sujay S. Parekh, Buğra Gedik, Henrique C. M. Andrade, Kun‐Lung Wu · 2010
Operator-based programming languages provide an effective development model for large scale stream processing applications. A stream processing application consists of many runtime deployable software processing elements (PE) that work in flows to process incoming messages. Operators (OP) are logical building blocks hosted by PEs. One or more OPs can be fused into a PE at compile-time. Performance optimization for our streaming system includes compile-time fusion optimization and runtime PE-to-host deployment. One of the goals of an optimized stream application is to use minimal computing resource to sustain maximal message throughput.