Serverless Outlier Management for Environmental IoT Data - A Case Study of PuhatuMonitoring
Shivananda R. Poojara, Argo Jõeleht, Pelle Jakovits, Satish Narayana Srirama · 2022
PuhatuMonitoring (PM) is an IoT application for observing the water level changes in a wetland in the Puhatu Nature Protection Area (NPA), North West Estonia next to an open-pit oil-shale quarry. IoT devices are installed for monitoring the environment and the LORAWAN things network stack is used to collect raw sensor data into the cloud. A major challenge faced by geologists were outliers in the collected data, caused by natural and weather conditions or device/sensor failures. We propose a system for data collection, detection of outliers using unsupervised machine learning algorithms, tagging of outlier data, and storing the results. We utilize Serverless (FaaS) model to deploy individual data processing services as virtual functions, which can be executed in an event-driven manner on data streams. We designed two different serverless data pipeline (SDP) approaches: Apache NiFi and Message Queue (MQTT) based. We investigated the performance of the two proposed approaches and results showed that Apache NiFi-based SDP approach has a lower processing time and higher disk utilization. In contrast, the MQTT-based approach experienced higher processing time and lower disk and memory utilization. Our suitability analysis showed that Apache NiFi-based approach suits better for environment monitoring applications like PuhatuMonitoring.