A method of cleaning data from IoT devices in Big data systems
Janusz Bobulski, Mariusz Kubanek · 2022 IEEE International Conference on Big Data (Big Data) · 2022
When retrieving data from IoT devices, errors in time series data often occur due to interference. Data with errors cannot be processed or saved in databases and warehouses because it causes data inconsistency, conflicting with Big Data principles. Built-in mechanisms in databases are not always able to fix incorrect data. Manually correcting data for extensive collections is too time-consuming and costly. Therefore, there is a need for automatic data cleansing, especially time series. This article proposes our data cleaning method for time series based on a moving average. Test results show a slight improvement in the signal-to-noise ratio.