Using Stream Processing for Real-Time Clock Drift Correction in Distributed Data Processing Systems
Roman Moravskyi, Ye. V. Levus · 2024
In large-scale smart device networks, such as those used in metering systems, the internal clocks of one-way communication devices can drift over time, leading to inaccuracies that impact data-driven decision-making and operational efficiency. Existing solutions often rely on postcollection corrections or synchronization methods that are not feasible for systems requiring real-time accuracy. This study evaluates a method for correcting time drift in smart devices using stream processing. By leveraging a distributed architecture with Apache Kafka, Flink, and OpenSearch, the proposed system processes a continuous stream of device time points at a high rate, ensuring accurate and scalable time correction. Benchmark tests demonstrate the system's capability to maintain low backpressure and high throughput under varying conditions. The findings suggest that the proposed approach is effective for real-time clock drift correction in large-scale networks, offering insights for future integration into Meter Data Management Systems (MDMS).