Design and Construction of the Data Warehouse Based on Hadoop Ecosystem at HLS-II
Yifan Song, Xin Chen, Chuan Li, Gongfa Liu, Wang, Ji-Gang, Xuan Ke · JACOW · 2019
A data warehouse based on Hadoop ecosystem is designed and constructed for Hefei Light Source II (HLS-II). The ETL program based on Spark migrates data to HDFS from RDB Channel Archiver and the EPICS Archiver Appliance continuously and store them in Parquet format. The distributed data analysis engine based on Impala greatly improves the performance of data retrieval and reduces the response time of queries. In this paper, we will describe our efforts and experience to use various open sources software and tools to effectively manage the big data. We will also report the plans on this data warehouse in the future.