Malware Detection Using Deep Learning and CNN Models
Marwa Ben Jabra, Omar Cheikhrouhou, Nesrine Atitallah, Anouar Ben Amor, Habib Hamam · 2023
The rise of cyberattacks has necessitated the development of effective malware detection mechanisms. Deep learning, with its ability to learn complex features from raw data, has been widely used for this purpose. This paper presents a two-fold contribution to the field of malware detection using deep learning. Firstly, we use seven pretrained CNN models to classify malware images from the Malimg dataset. While these models achieved high accuracy, they presents low precision and F1-score, indicating that they were prone to false positives and false negatives. Additionally, these pre-trained models are susceptible to overfitting, which is a common issue with transfer learning. To overcome this limitation, we propose a custom CNN model consisting of six layers, trained from scratch on the Malimg dataset. To address potential issues like over-fitting and data imbalance when training models from scratch, we used regularization techniques such as L1 or L2 regularization, dropout. Our proposed custom CNN model outperformed the pretrained models, achieving best/average accuracy values over 5 trials of 100%/98.26%. We also found that our custom model was less susceptible to over-fitting and adversarial attacks. Our proposed approach provides a promising solution to the problem of limitations in using pre-trained models for malware detection.