Efficient Concept Drift Detection: A Meta Feature Selection Approach
Zelong Liu, Pingfan Wang, Nanlin Jin · 2024
Concept drift in data streams significantly impacts predictive modelling, as the underlying distribution of the training sample evolves, often leading to increased error rates and degraded model performance. When dealing with big data which has many features and data speed is high, traditional concept drift detection methods might delay in analyzing and detecting in time. This paper introduces an improved drift detection model utilizing multivariate analysis methods for feature selection, thereby enhancing the model's ability to detect drift more efficiently. Our approach analysis various parameters of the data stream to select the most important features and conduct conduct drift detection based on the selected features. Experimental results demonstrate that this feature-selected drift detection model not only maintains classification performance, but also significantly reducing computational overhead. This predictive framework is particularly valuable in scenarios where large data streams require real-time analysis and where computational resources are limited, providing a practical solution for maintaining robust model performance in dynamic environments.