Relational Data Enrichment by Discovery and Transformation
Fatemeh Nargesian · TSpace (University of Toronto) · 2019
In the context of data preparation for data science, we study the data enrichment problem, which is the challenge of augmenting a table with relevant data. We consider two enrichment paradigms: (1) data enrichment via discovery, and (2) data enrichment via transformation. In the first paradigm a table is enriched with relevant data discovered in data lakes. Specifically, we study table union search as the search problem of finding tables that can be unioned with a query table. We present a probabilistic solution for finding top-k unionable tables with a query table within massive data lakes. We show that our table union search outperforms in speed and accuracy existing algorithms for finding related tables and scales to provide efficient search over large open data lakes. We also consider the complementary problem of building a directory structure over a data lake. We define a data lake organization as a navigation structure (graph) containing nodes representing sets of attributes within a data lake. We present a new probabilistic model of how users navigate a data lake and propose an approximate algorithm for constructing the optimal directory structure. We show that the organizations constructed by this algorithm dramatically improve the expected probability of discovering tables over a baseline technique and linkage graphs. In the second paradigm, a table is enriched with new attributes (features) that are constructed by transforming the existing attributes in the table. We define the prediction-based feature engineering for classification as the problem of predicting the most effective feature to augment a table with. We show that we can enrich tables with new attributes that improve the performance of classification tasks. We also show that the prediction-based feature engineering technique outperforms other feature engineering approaches in effectiveness and response time.