Query Processing for Heterogeneous Data Sources in SQL Generator
N. I. Pikuleva, A. Sh. Khafizova, R.A. Khakov · 2020 International Multi-Conference on Industrial Engineering and Modern Technologies (FarEastCon) · 2020
There is a big data amount, which should be analyzed, but it is stored on different data sources. This paper describes the solution for query processing on heterogeneous data sources. Research concentrates on analyzing solutions for aggregating data from different data structures (structured, unstructured and semi-structured data) using SQL queries. Functional and stress testing of solutions is carried out using this data. Finally, it is introduced the extension module for Grizzly Framework relieves working with different sources using PostgreSQL FDW technology. This module allows to generate queries to create third-party tables for CSV files (also remote files), relational databases (PostgreSQL, MS SQL Server, MySQL, SQLite, Oracle), non-relational databases (MongoDB and Neo4J), HDFS distributed storages. Developers can still focus on solving their problem without storing queried data in memory.