Feature Selection on Big Data using Masked Sparse Bottleneck Centroid-Encoder
Tomojit Ghosh, Michael Kirby · 2023
We introduce a nonlinear model, Masked Sparse Bottleneck Centroid-Encoder (MSBCE), for determining the features that discriminate between two or more classes. The algorithm aims to extract discriminatory features in groups while reconstructing the class centroids in the ambient space and simultaneously use additional penalty terms in the bottleneck layer to decrease within-class scatter and increase the separation of different class centroids. The model has a sparsity-promoting layer (SPL) with a one-to-one connection to the input layer. Along with the primary objective, we minimize the $l_{2,1}$-norm of the sparse layer, which filters out unnecessary features from input data. During training, we update class centroids by taking the Hadamard product of the centroids and weights of the sparse layer, thus masking the irrelevant features from the target. Therefore the proposed method learns to reconstruct the critical features of the class centroids. The algorithm is applied to various real-world data sets, including high-dimensional biological, image, speech, and accelerometer sensor data. We compared our method to different state-of-the-art feature selection techniques, including supervised Concrete Autoencoders (SCAE), Feature Selection Networks (FsNet), Stochastic Gates (STG), and LassoNet. We empirically showed that MSBCE features often produced better classification accuracy than other methods on the sequester test sets, setting new state-of-the-art results. Apart from achieving state-of-the-art results, this is the first time feature selection is done on two high-energy particle physics data sets, SUSY and HIGGS, with 4.5 million and 10 million samples, respectively. The selected features, which are only 25-30% of the total number of variables, attained almost similar prediction rates compared to all features.