A Comprehensive Approach to Real-Time and Batch Processing for Energy-Efficient IoT Homes: Leveraging Lambda Architecture and Data Lakes
Filippos Serepas, Ioannis Papias, Nikolaos Bellos, Vangelis Marinakis · 2024
The incorporation of IoT technology into energy-efficient home systems has resulted in a surge in data volume, prompting the need for sophisticated storage and processing solutions. This paper proposes a system that integrates the Lambda Architecture with data lakes to address real-time and batch processing needs in the context of energy-efficient homes. By leveraging technologies such as TimescaleDB for short-term storage and Apache Hudi for long-term storage, coupled with Kafka for data streaming, the system ensures efficient data management and analysis. Real-time insights are provided through GraphQL-powered visualizations, while batch processing facilitates advanced analytics and machine learning model training. The proposed system addresses the dual demands of end-users seeking real-time insights and data scientists requiring extensive datasets for analysis.