Tiny-Impute: A Framework for On-device Data Quality Validation, Hybrid Anomaly Detection, and Data Imputation at the Edge
Shamil Al-Ameen, Bharath Sudharsan, Tomasz Szydło, Roua Al-Taie, Tejal Shah, Rajiv Kumar Ranjan · 2023
In the landscape of Internet of Things (IoT) systems, data quality degradation can occur continuously for several reasons, such as sensor malfunctions, intermittent network availability, device maintenance, or incomplete data collection. This paper proposes three efficient data quality validation and imputation algorithms that identify and replace noisy and missing values with better-quality data. We extensively stress-tested and evaluated our algorithms by deploying them on microcontrollers and small CPU-based IoT boards with memory limited to as little as 32KB. We simulated a sensor data stream using five real-world datasets, including data collected from the Newcastle Urban Observatory. In this setup, each algorithm excelled in different areas, consistently demonstrating high performance across 100 samples in terms of high energy efficiency (0.014 J), low computation time (85.94 ms), and low error rates (0.0019 MAE, 0.0027 RMSE). Remarkably, we found that on average, our algorithms running on hardware, costing less than $10, showed performance on par with state-of-the-art methods on high-end devices. The results also demonstrated that our algorithms enabled on-device cleansing of live streaming sensor data, eliminating the dependency on cloud services and allowing for real-time data quality validation and processing at the edge.