Research on Distributed Storage and Query Optimization of Multi-source Heterogeneous Meteorological Data
Xiaodong Hu, Huanli Xu, Jinfang Jia, Xiaoying Wang · 2018
The growth of massive data makes the real-time data service of meteorological forecast and climate analysis facing severe challenge. Distributed database is a good solution to meet the needs for massive multi-source heterogeneous meteorological data storage. Since the current mainstream HBase database fails to support the non-Rowkey query, the poor performance of the real-time query of meteorological data is unsatisfactory. To address this issue, three kinds of distributed data query optimization strategies are proposed in this paper, including query optimization based on secondary index, secondary index query optimization based on hotscore, and query optimization based on the Redis hot data caching strategy. The corresponding experimental results indicate that the search scheme based on the Redis hot data caching strategy has the best performance among the three schemes, not only can meet the needs of meteorological service query, but also can achieve 3-8 times efficiency enhancement than standard HBase.