Adapting State-Intensive Non-Blocking Queries over Distributed Environments
Bin Liu, Elke Angelika Rundensteiner · 2005
Main memory is a critical resource in push-based non-blocking query processing, especially for queries with stateful operators. Works in the literature apply partitioned parallel processing to alleviate the stringent memory demands. However, main memory of a distributed system remains limited. Thus, there is a demand for efficient main memory usage even for partitioned parallel queries. In this work, we first investigate two adaptations, namely, disk-based adaptation and distributed adaptation, that adapt operator states when memory overflow happens for complex multi-input operators. We analyze the tradeoffs regarding the factors and polices to be used when adapting operator states to overcome memory overflow. Two approaches, namely, lazy-disk and active-disk adaptations, are proposed to integrate the disk-based and distributed adaptations when the aggregated main memory of a distributed system is not sufficient for the query processing. Both appproaches aim to maximize the overall throughput. Extensive experiments have been conducted on a working system. These experiments reveal various aspects of partitioned parallel processing and their adaptation strategies.