Storing Intermediate Results in Space and Time: SQL Graphs and Block Referencing
Basem Elazzabi · 2021
With the advancement of data-collection technology and with more data being available for data analysts for data-intensive decision making, many data analysts use client-based data-analysis environments to analyze that data. Client-based environments where only a personal computer or a laptop is used to perform data analysis tasks are common. In such client-based environments, multiple tools and systems are typically needed to accomplish data-analysis tasks. Stand-alone systems such as spreadsheets, R, Matlab, and Tableau are usually easy to use, and they are designed for the typical, non-technical data analyst. However, these systems are limited in their data-analysis capabilities. More complex data analysis systems provide more powerful capabilities, such as database management systems (DBMSs). However, these systems are complex to use for the typical data analyst and they specialize in handling a specific category of tasks. For example, DBMSs specialize in data manipulation and storage but they do not handle data visualization. As a consequence, the data analyst is usually forced to use multiple tools and systems to be able to accomplish a single data-analysis task. The more complex and demanding the data-analysis task is, the more tools and systems are typically needed to complete the task. One monolithic data-analysis system cannot satisfy all data-analysis needs. Embracing diversity, where each tool and system specializes in a specific area, allows us to satisfy more needs than a monolithic system could. For example, some tools can handle data manipulation, while others handle different types of visualizations. However, these tools typically do not interoperate, requiring the user to move data back and forth between them. The result is a significant amount of time wasted on extracting, converting, reformatting, and moving data. It would help to have a common client-side data platform that the data-analysis tools can all use to share their results, final and intermediate. Sharing intermediate results is especially important to allow the individual data-analysis steps to be inspected by a variety of tools. Moreover, sharing intermediate results can eliminate wasted computations by building on top of previous results instead of recomputing them, which can speed up the analysis process. In this research we explore a new data paradigm and data model that allows us to build a shared data-manipulation system for a client-based data-analysis environment. In this shared system, we factor out the data manipulation process from data-analysis systems and tools (the front-end applications) into the shared system, leaving the front-end systems and