Communication cost optimization for cloud Data Warehouse queries
Swathi Kurunji, Tingjian Ge, Benyuan Liu, Cindy X. Chen · 2012
Read-Optimized databases are well suited for read intensive Data Warehouse applications. In addition, data in these applications grow rapidly and hence need a dynamically scalable environment like Cloud. Cloud provides a flexible environment where user can load data, execute queries and scale resources on demand. However, cloud has its own challenges. To reduce the inter-node communication during the execution of query, tables are horizontally partitioned on join attribute and then related partitions are stored on the same physical system. In cloud environment it is not possible to ensure that these related partitions are always stored on the same physical system. As the resources are scaled up, the number of nodes involved increases, resulting in the increased inter-node communication. This becomes critical when we have huge data (in Tera or Peta bytes) stored across a large number of nodes. So with the increase in number of nodes and data size, the communication message size increases. All these factors result in increased bandwidth usage and performance degradation. When the number of joins in a query increases, the performance will further degrade. These problems emphasize a need for good storage structure and query execution plan. In this paper we propose a storage structure PK-map and a query processing algorithm. We show, through experiments, that this approach not only decreases the inter-node communication overhead but also decreases the work load of joins.