Machine Learning-Powered Malware Detection in Encrypted IoT Traffic
Arshad Farhad, Muhammad Irfan Khan, Ali Hassan Sodhro, Muhammad Khurram Ehsan, Fatiha Djebbar · 2025
The exponential growth of encrypted network traffic in IoT ecosystems has created a critical challenge: maintaining privacy while enabling effective malware detection. This paper presents a machine learning (ML) and deep learning (DL) framework for detecting sophisticated malware (e.g., ransomware, trojans, and spyware) in encrypted Internet of Things (IoT) traffic, combining feature engineering with model fusion techniques. We evaluate Random Forest, LSTM, and RNN models on the CIC MalMem 2022 dataset, achieving an 87.1% accuracy with Random Forest - significantly outperforming sequential models (74.6%). Our proposed methodology includes: (1) a novel feature selection pipeline using mutual information for encrypted IoT traffic analysis, and (2) comprehensive benchmarking of traditional ML versus DL approaches. The results demonstrate this framework can potentially be deployed in smart cities and healthcare IoT systems where encrypted traffic analysis must balance detection accuracy with computational efficiency.