Raising the Parallel Abstraction Level for Streaming Analytics Applications
Gabriele Mencagli, Massimo Torquati, Dalvan Griebler, Marco Danelutto, Luiz Gustavo Fernandes · IEEE Access · 2019
In the stream processing domain, applications are represented by graphs of operators arbitrarily connected and filled with their business logic code. The APIs of existing Stream Processing Systems (SPSs) ease the development of transformations that recur in the streaming practice (e.g., filtering, aggregation and joins). In contrast, theirparallelism abstractionsare quite limited since they provide support to stateless operators only, or when the state is organized in a set of key-value pairs. This paper presents how theparallel patternsmethodology can be revisited for sliding-window streaming analytics. Our vision fosters a design process of the application ascompositionandnestingof ready-to-use patterns provided through a C++17fluentinterface. Our prototype implements the run-time system of the patterns in the FastFlow parallel library expressingthread-based parallelism. The experimental analysis shows interesting outcomes. First, our pattern-based approach allows easy prototyping of different versions of the application, and the programmer can leverage nesting of patterns to increase performance (up to 37% in one of the two considered test-bed cases). Second, our FastFlow implementation outperforms (three times faster) the handmade porting of our patterns in popular JVM-based SPSs. Finally, in the concluding part of this paper, we explore the use of atask-basedrun-time system, by deriving interesting insights into how to make our patterns library suitable for multi backends.