Advanced Network Malware Detection: Integrating PySpark and Machine Learning Techniques
Mostafa Dorrah, Asmaa ElMaghraby, Abdallah ElSaadany, Mohamed Atta, Ahmed Ashraf, Yousef Adel, Tawfik Yasser, M Fathi, Ibrahim Abdelbaky · 2024
This study introduces a novel approach for detecting malware using cutting-edge machine Learning with PySpark, which targets the increasingly complex cyber threats that conventional security measures struggle to counter. Our research aims to create a highly accurate system for detecting harmful network activities by examining network traffic patterns, including the duration and frequency of connections, the amount of data transferred, and the involved endpoints. We utilized a comprehensive dataset from Kaggle and performed thorough data preprocessing and feature selection to optimize our models' performance. The research employed a variety of algorithms such as Logistic Regression, Random Forest, Decision Tree, and Naive Bayes. Each model was fine-tuned and evaluated based on its recall in distinguishing between safe and dangerous software. Our work seeks to make a significant impact in the field of cybersecurity by presenting an effective machine learning-driven tool for accurate malware detection, thereby improving network security.