Reinforcement learning-driven feature selection for enhanced classification in cybersecurity: Applications in IoT security and malware detection
Hanaa Fathi, Ola Malkawi, Arar Al Tawil, Amneh Shaban, Dyala Rasheed Ibrahim, Mohammad Adnan Aladaileh · International Journal of Data and Network Science · 2025
The effectiveness and efficiency of a machine learning model can be improved by feature selection, especially for high-dimensional datasets such as in cybersecurity. The proposed approach utilizes an enhanced version of the Rainbow agent with a memory storage structure. The suggested approach is assessed using two benchmark datasets namely RT-IoT2022 which is targeted towards IoT network security and the Android Malware Detection dataset which is meant for mobile security. The specification of the reinforcement learning model has been trained for 20 epochs and it is progressively enhanced through feature subsets to enhance classification accuracy. The results show that the AUC scores continuously increase were the one for RT-IoT2022 achieves 0.91 and Android at 0.93. Three well-known classifiers XGBoost, Random Forest and multi-layer perceptron (MLP) are used to test the power of the selected features. The outcome evaluation on RT-IoT2022 dataset shows that Random Forest achieved maximum accuracy (99.48%), followed by XGBoost (99.16%), while MLP secured 94.04% accuracy. In the Android malware dataset, XGBoost model gave the best accuracy of 89.50%, followed closely by Random Forest with 87.00% and MLP with 86.50%. This clearly shows that reinforcement learning based feature selection enhances accuracy and reduces computation. The research emphasizes utilizing dynamic feature selection in any cyber security application. The future will experiment with incorporating deep reinforcement learning as well as hybrid selection.