An Online Approximate Stream Processing Framework with Customized Error Control
Xiaohui Wei, Yuanyuan Liu, Xingwang Wang, Shang Bing Gao · 2018
In online approximate stream processing, customers generally submit their requests with some specific quality requirements (e.g. maximum error). This raises a critical problem that online quality control is necessary to meet customized requirements. Since continuous arriving data needs to be processed immediately, it brings the difficulty of acquiring knowledge which significantly affects the efficiency of sampling. Hence, it's more challenging to ensure a prescribed level of quality without knowledge about data. In this paper, we present an adaptive approximate processing framework for online stream applications to address the challenges mentioned above. Specially, we first design a new data knowledge learning scheme to stratify the arriving stream data. Then, based on the online learning results, we propose a dynamic sampling strategy with the consideration of the stream rate. Finally, we further present a double-check error control mechanism to manage the output quality. Experiments with real world datasets show that the proposed approximate framework is not only applicable to different data distributions, but also provides a customized error control.