uitPETransMDS: A PE Malware Detection System Using a Hybrid Approach of Transfer Learning and Image Visualization
Trinh Gia Huy, Luong Nguyen Thanh Nhan, Nguyen Tan Cam · 2024
The rapid expansion of devices and computers, alongside technological advancements, has led to a corresponding increase in malicious attacks. Malware attacks represent a significant global threat in cyberspace. Despite the concerted efforts of security researchers and security companies to mitigate these attacks, they remain a persistent challenge. This has prompted a substantial focus on research endeavors to combat malicious software. Recent advancements in artificial intelligence have spurred various research efforts in malware detection. This paper focuses on detecting and classifying malware using transfer learning models, such as variants of VGG, ResNet, and MobileNet, on a grayscale image-based PE dataset with dimensions of$64 \mathrm{x}64 \mathrm{x}3$pixels. The proposed architecture comprises three modules. The first module is responsible for converting the PE binary file to a grayscale image. The second module aims to identify whether the input file is malicious or benign. Among the models tested, MobileNetV1 achieved the highest performance with an accuracy of 98.70%and an Fl-score of 96.46%. The third module, utilizing the ResNet101 model, is responsible for malware classification (five different types in the dataset). It achieved an accuracy of 98.78% and an Fl-score of 97.52% on the testing set.