Identification and Processing of Outliers in Data Based on Improved Quantile Method
Xiong Wei, Wanyuan Nie, Zhenyu Wu, Luo Zhihong · 2019
At present, a variety of sensors were installed on various production equipments, and a large amount of data was collected by sensors during operation. Then, it is inevitable that there are a large number of outliers in the measured values. In view of the above situation, this article has studied common methods in data outlier filtering. Pauta Criterion, Chauvenet's Criterion and Quantile Method were analyzed. The shortcomings of these methods and the defects in the application were summarized, and according to the actual problem, Quantile Method has been improved in a targeted manner. The advantages and effectiveness of the proposed method were proved by simulation experiments. The experimental results shown that the improved Quantile Method has higher recognition accuracy of outliers than the other two methods, and the new method is suitable for data with large fluctuations. Compared with the other two methods, it has higher practical application value. After the outliers in the data were identified, the average method was used for further correction. The method proposed in this paper has fast calculation speed and high accuracy, can adapt to the data with large fluctuations, and can accurately correct the outliers according to the fluctuation trend of the data. In summary, the method is suitable for real-time filtering of data during data collection.