An adaptive data cleaning framework: a case study of the water quality monitoring system in China
Zeng Chen, Peng Jiang, Jun Liu, Song Zheng, Zhenyu Shan, Zhihua Li, Huan Xu, Igor Vyacheslavovich Bychkov, Alexei E. Hmelnov · Hydrological Sciences Journal · 2022
Robust detection of patterns at low signal-to-noise ratios (SNRs) is a fundamental challenge of analysing high-frequency data, particularly in water quality monitoring. When conducting water quality data analysis, an indispensable step is to clean the data noise. In this paper, a new method named ADAPTIVE-EWT-MFE, based on empirical wavelet transform (EWT) and multiscale fuzzy entropy (MFE), is proposed to implement time series data cleaning. EWT-MFE can decompose the spectrum into different intrinsic mode functions (IMFs). According to different characteristics of the IMFs, an adaptive and adjustable parameter based on MFE, which reflects the intrinsic characteristics, is introduced into the threshold function to improve the performance of noise cleaning. Finally, this hybrid data cleaning method is designed to filter the high-frequency noise on the IMFs. Results show that our proposed method not only has great potential to improve the noise-cleaning performance but also does not distort noise-free data.