Evaluating and Evading Machine Learning Malware Detectors with Deep Q Learning
K. Preetha, M. Dhilsath Fathima · 2025
Traditional machine learning-based malware detectors often rely on static features such as API calls, DLL imports, and PE header values. However, these systems remain vulnerable to adversarial evasion techniques that subtly modify malware without altering its functionality. In this work, we address the critical issue of classifier vulnerability by proposing a reinforcement learning-based adversarial evasion framework. A Random Forest classifier is first trained on a structured dataset of labeled malware and benign samples. To assess and exploit weaknesses in the classifier, we design a Deep Q-Learning (DQL) agent integrated into a custom OpenAI Gym environment that simulates adversarial behavior. The agent iteratively modifies binary feature vectors of malware samples, guided by a dynamic reward function based on the classifier’s confidence scores, aiming to achieve misclassification with minimal feature changes. Our results show that the DQL agent achieves a high evasion success rate, significantly reducing classifier accuracy on perturbed samples, while preserving malware behavior. The proposed framework highlights the need for robust defense mechanisms in static malware detection and serves as a testbed for evaluating classifier resilience against adaptive adversaries.