A Novel Framework for Concept Drift Detection using Autoencoders for Classification Problems in Data Streams
Usman Ali, Tariq Mahmood · 2022
In streaming data environments like weather forecasting, health care monitoring, network traffic monitoring and energy consumption etc., data characteristics and probability distributions are likely to change over time, posing challenges for classification models to predict accurately. In such non-stationary environments where the patterns in data can change quickly, the pre-trained machine learning models may become outdated and hence, there is a need to update the model to maintain an acceptable predictive performance. Existing approaches to drift detection, particularly supervised and unsupervised, have inherent problems. Supervised methods detect drift based on the error rate and assume that labels are available immediately after prediction which may not be possible in several real-world scenarios. Unsupervised methods on the other hand suffer from high rate of false alarms and curse of dimensionality (i.e., the complexity faced in detecting drift across an increased number of features). To develop a more applicable technology, in this paper, we are concerned with unsupervised drift detection i.e., to detect drift independent of the labels. A recent set of experiments has revealed that it is possible to make an autoencoder learn the data distribution of a pre-defined set of classes and distinguish between the different class samples as they arrive. Based on this, we propose an Autoencoder-based Drift Detection Method (AE-DDM) which monitors the distribution of reconstruction loss in an incoming batch stream based on a thresholding mechanism to generate warnings and detect drift. AE-DDM has been tested on four synthetics (RBM, Hyperplane, Stagger and Gaussian)and one real word dataset (NOAA) with sudden and gradual drifts. AE-DDM starts generating warnings and then conforms the drift with zero delays in case of sudden drift with reduced false alarms. The classification results of ten most used machine learning classifiers on the NOAA dataset show that the detected drift is real. In this context, the potential contributions of our research are: 1) a novel framework for unsupervised drift detection based on the deep learning autoencoder technology, 2) A brief comparison of available drift detection approaches with proposed properties of an ideal drift detector, c) creation of a synthetic Gaussian dataset for drift detection research, and d) use of count threshold in conjunction with a batch threshold to combat false alarms in drift detection.