Data Stream Processing on Embedded Devices
René Müller · Repository for Publications and Research Data (ETH Zurich) · 2010
Over the years, online processing of data has become more important.In inventory management, monitoring and financial applications data is generated in streams that are processed on the fly instead of being stored in a repository and processed offline later.As data volumes increase, processing has to be offloaded from the central stream processing engine.Processing either has to be moved to the data sources or to specialized accelerators placed between data source and engine.Stream processing platforms thus become heterogeneous.The problem is how to optimize query execution when it spans different streaming systems.This thesis discusses stream processing on two different platforms: wireless sensor networks and field-programmable gate arrays (FPGAs).Both are typically connected to traditional server-class streaming processors.Driven by the different optimization goals (e.g., throughput, latency, resource consumption) operators have to be carefully placed.In some cases it is more efficient to place operators, for example, into the sensor network.Other operators that require a substantial amount of state are better placed on the server.The problems addressed in this work is the design of the underlying execution platforms that facilitate the operator placement, strategies, and cost-models used for optimization.In the first part of the thesis SwissQM is presented.SwissQM is a stream processing platform for wireless sensor networks and is based on a small virtual machine that is deployed on resource-constrained sensor nodes.Declarative queries submitted by the user are translated into short bytecode sequences and are disseminated in the network.The bytecode programs implement streaming operators that are executed at the data source in the network.The remaining operators of the queries are placed onto the base station that connects the wireless sensor network to, e.g., the Internet.The base station also performs multi-query optimization of multiple user queries that are executed concurrently.The thesis proposes an energy-based cost model and presents optimization strategies that rely on the cost model.Multi-query optimization maximizes utilization of the network infrastructure such that expensive deployments can be accessed by several users and applications concurrently.The second part of the thesis applies the same techniques to FPGAs, i.e., the automated compilation of queries into digital circuits that can be placed onto iii