Scalable and efficient active service integration
David Maier, Paul Benninghoff · 2003
We investigate issues in the construction of an Active Service Integration System (ASIS). An ASIS is a mediation system that provides event-based monitoring and integration over data-intensive, networked services. We describe the design and implementation of the Paradox Active Service Integration System, which we have built as part of our dissertation work. Paradox addresses many of the fundamental issues of ASIS construction. Paradox extends database technology and previous work in data integration to handle event-based processing over autonomous, heterogeneous network services in an efficient manner. We suggest capabilities and metadata that services may provide to aid in the active integration process. Efficiency and scalability problems remain in Paradox that must be addressed in a practical system. We describe and evaluate methods for addressing several such problems that are unified by two central themes: inter-task sharing, and the specification and exploitation of increasingly rich service characteristics. Data caching is essential to a scalable and efficient ASIS. We describe how the long-lived nature of ASIS requests can be leveraged to make effective caching decisions. We present a detailed model and framework for effective, cost-based selection of a view cache in an ASIS. ASIS cache selection involves multiple complex tasks: the selection of the view cache, the optimization sub-problems that involve multiple, simultaneously-executing queries (MQO subproblems), and the generation of efficient plans that incorporate and maintain the chosen view cache. The resultant optimization problem is doubly-exponential in complexity. We describe a multi-pronged approach to handling this problem in a tractable manner. We present a description and implementation of Multiplex Query Optimization (MuxQO), a novel method for efficiently handling problems that can be cast as a group of overlapping query optimization problems. We characterize the applicability of MuxQO, and we describe a performance evaluation that demonstrates the effectiveness of MuxQO in handling ASIS view selection. MuxQO handles cache selection, MQO subproblems, and optimal plan generation in an integrated fashion. We describe how a top-down optimizer can be modified to support MuxQO. MuxQO is applicable to a range of problems, including physical database design, multiple query optimization, evaluation of recursive queries, and the physical representation of new data formats and models such as XML. Finally, we describe and evaluate a novel approach to exploiting rich application semantics to improve the efficiency and scalability of an ASIS. In particular, we describe a method for exploiting constraints on information change over time. Our approach can greatly improve the scalability of an ASIS with respect to the frequency of change events at component sources. We argue that application-level semantics are a rich vein to mine in improving the scalability and efficiency of active service integration.