A Tutorial on IoT Streaming Data Pipelines
Qusay F. Hassan, Neha Pandey · 2025
The emergence of the Internet of Things (IoT) has transformed how data are collected and processed. IoT applications often generate massive amounts of continuous, high-velocity, and heterogeneous data. In industrial settings, the timely capture and analysis of such data is crucial for monitoring and optimizing operations. As a result, many organizations today integrate streaming data pipelines into their projects to enable fast and smart decision-making. This chapter provides a comprehensive tutorial on streaming pipelines and how they can be integrated into IoT environments. It covers the fundamentals of these systems and explains how organizations use them. It then introduces key open-source technologies widely used in building modern IoT streaming pipelines: Node-RED, MQ Telemetry Transport (MQTT), Apache Kafka, and Apache Flink. The chapter continues with a hands-on demonstration of constructing a robust IoT streaming pipeline using these technologies in a cloud-based environment to collect, process, and store Industrial Internet of Things (IIoT) data in real time. The detailed discussions and step-by-step guidelines provided will help researchers and practitioners effectively implement and integrate these systems into their IoT projects.