Dimensionality Reduction Strategies for Classification: ML Versus DL Approaches and Their Combinations
Chihli Hung, Chih‐Fong Tsai, Ming‐Hui Wu · Expert Systems · 2025
ABSTRACT Dimensionality reduction plays a vital role in enhancing the performance of data classification tasks by reducing the complexity of the feature space. This study examines the effectiveness of integrating dimensionality reduction techniques with classification algorithms across four strategic configurations: (1) machine learning (ML)‐based dimensionality reduction with ML classifiers, (2) deep learning (DL)‐based dimensionality reduction with DL classifiers, and two heterogeneous combinations that mix ML and DL methods. Using 20 benchmark datasets from diverse domains, with feature dimensions ranging from 44 to 19,993, we systematically evaluate and compare these configurations. The dimensionality reduction methods include three ML‐based feature selection techniques, Genetic Algorithm (GA), Information Gain (IG), and the C4.5 decision tree, and four DL‐based feature extraction approaches, Autoencoder (AE), Sparse Autoencoder (SAE), Denoising Autoencoder (DAE), and Variational Autoencoder (VAE). For classification, Support Vector Machine (SVM) and k‐Nearest Neighbours (KNN) are used as ML classifiers, while Multilayer Perceptron (MLP) and Deep Belief Network (DBN) serve as DL classifiers. Experimental results show that SAE consistently produces the most compact feature sets and improves classification performance, with the SAE + MLP combination achieving the best overall results. Furthermore, we explore ensemble dimensionality reduction strategies that integrate multiple algorithms. Although the best ensemble approach slightly outperforms the SAE + MLP model, the observed performance improvements are not statistically significant, that is, 0.839 versus 0.836 for AUC rates. In addition, SAE achieves a significantly higher dimensionality reduction rate compared to the best ensemble method (63% vs. 18%).