Research on environmental sensor data analysis based on real-time data warehouse
Taizhi Lv, Yujuan Zou, Yaowei Wang, Enze Wu · 2024
Environmental sensors play a crucial role in modern society. They provide strong support for environmental protection, resource management, and sustainable development by monitoring and perceiving various parameters in real-time. With the development of big data and artificial intelligence technologies, intelligent management of environmental data has become possible. To enhance the level of intelligence in environmental monitoring, this paper studies the performance of environmental sensor data operations in data warehouses such as Hive, ClickHouse, and Doris, and constructs an environmental sensor data analysis system based on a real-time data warehouse. The system employs Flink for real-time collection of environmental data, ClickHouse for data storage, Spark for data analysis, and front-end and back-end technologies for data visualization. The system achieves automatic collection, processing, and analysis of environmental data, thereby improving the efficiency and accuracy of environmental monitoring.