Advancing the design and implementation of Big Data Warehousing Systems
Carlos Costa · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · 2019
Current Information Technology advancements have led organizations to pursue high business value and competitive advantages through the collection, storage, processing, and analysis of vast amounts of heterogonous data, generated at ever-growing rates. Since a Data Warehouse (DW) is one of the most remarkable and fundamental enterprise data assets, nowadays, a current research trend is the concept of Big Data Warehouse (BDW), characterizing real-time, scalable, and high-performance systems with flexible storage based on commodity hardware, which can overcome the limitations of traditional DWs to assure mixed and complex Big Data analytics workloads. The state-of-the-art in Big Data Warehousing (BDWing) reflects the young age of the concept, as well as the ambiguity and lack of integrated approaches for designing and implementing these systems. Fulfilling this gap is of major relevance, reason why this work proposes an approach composed of several models and methods for the design and implementation of BDWs, focusing on the logical components, data flows, technological infrastructure, data modeling, and data Collection, Preparation, and Enrichment (CPE). To demonstrate the usefulness, effectiveness, and efficiency of the proposed approach, this work considers four demonstration cases: 1) the application of the proposed data modeling method in several potential real-world applications, including retail, manufacturing, finance, software development, sensor-based systems, and worldwide news and events; 2) the application of the CPE method to process batch and streaming data arriving at the BDW from several source systems; 3) a custom-made extension of the Star Schema Benchmark (SSB), named the SSB+, in which several workloads were developed to benchmark a BDW implemented using the proposed approach, comparing its performance against a traditional dimensional DW; 4) a real-world instantiation based on the development of a BDWing system in the context of smart cities. The results of this research work reveal that the approach can be applied and generalized to support several application contexts, providing adequate and flexible data models that can reduce the implementation time between data collection and data analysis. Moreover, the proposed approach frequently presents faster query execution times and more efficient resource usage than a traditional dimensional modeling approach. Consequently, the proposed approach is able to provide general models and methods that can be used to design and implement BDWs, advancing the state-of-the-art based on a systematic approach rather than an ad hoc and use case driven one, which is seen as a valuable contribution to the technical and scientific community related to this research topic.