Real-Time Data Processing Architectures for IoT Applications: A Comprehensive Review
Siddhi Dingorkar, Shekhar Kalshetti, Yukta Shah, Prashant Lahane · 2024
The advent of the Internet of Things (IoT) has resulted in an exponential surge in data generation, necessitating sophisticated data engineering solutions for real-time processing. This review paper scrutinizes the challenges associated with managing the vast volumes, diversity, and velocity of IoT data. It underscores the significance of low-latency processing and emphasizes the imperative for robust security and privacy measures. Key strategies elucidated encompass edge computing for latency reduction and bandwidth optimization, alongside the utilization of stream processing frameworks such as Apache Kafka and Apache Flink for real-time analytics. Furthermore, the paper delves into the realm of distributed computing, particularly Apache Spark, and investigates the integration of machine learning to augment decision-making and glean insights from IoT data streams.