Unveiling the Role of Diversity Measure in Concept Drift Detection
Osama A. Mahdi, Nawfal Ali, Eric Pardede, Bhagwan Das · Advances in systems analysis, software engineering, and high performance computing book series · 2025
Concept drift, marked by changes in the statistical properties of a target variable, is a notable challenge in machine learning, data mining and applications involving big data and large-scale data processing. Addressing this, the employment of diversity measure has emerged as an effective strategy. This chapter examine and investigate the role of the diversity measure in detecting concept drift and compare and analysis four different ways of using them: DMDDM for drift` detection in a fully supervised binary classification context, DMDDM-S in a semi-supervised context, DMODD for online drift detection in a fully supervised multi-classification context, and HBBE, a hybrid block-based ensemble designed for addressing different types of concept drifts. The comparative analysis evaluates the efficacy of these methods and the results confirm their effectiveness within their respective settings. Overall, this chapter explores advancements in using diversity measures for concept drift detection in big data, emphasizing their importance and future research in machine learning contexts.