OSK: Optimal Subsampling Method Based on K-means Clustering for Imbalanced Big Data
Lili Li, Heng Xiao, Yu Wang, Haolun Shi, Jiguo Cao · Research Square · 2023
Abstract While existing methodologies have effectively addressed big data subsampling, there is a noticeable gap in research concerning imbalanced subsampled data. To tackle the unique challenge of low specificity in the minority class within highly imbalanced datasets, we introduce the Optimal Subsampling technique rooted in K-means clustering (OSK). This method is specifically designed for massive yet imbalanced datasets. Our proposed approach employs an optimal subsampling mechanism to extract representative subsamples and utilizes K-means clustering to transform imbal-anced subsamples into multiple balanced datasets. In the spirit of ensemble learning, the OSK method generates predictions by averaging predicted values from multiple models and subsequently derives categorical decisions. The effectiveness of the OSK method is substantiated through extensive simulation studies and a real-world data example. Its superiority is evident in terms of shorter running time and higher classification accuracy when compared to existing methods.