Scalable Intrusion Detection in IoT Networks: Evaluating PySpark Pipelines and Design Trade-Offs
Michael Georgiades, Faisal Hussain, Lakis Christodoulou, Kin-Hon Ho, Yun Hou, Andreas Gregoriades · 2025
The rapid growth of Internet of Things (IoT) networks has introduced challenges in securing large-scale, real-time environments against evolving cyber threats. This study evaluates scalable machine learning workflows implemented in PySpark for intrusion detection using the RT-IoT2022 dataset. We compare manual feature engineering with automated pipeline-based approaches across classifiers including Logistic Regression, Naïve Bayes, Decision Tree, and Random Forest. Leveraging PySpark's distributed processing and modular components—such as Pipeline, StringIndexer, VectorAssembler, and MinMaxS-caler—we assess how workflow design affects performance metrics (Accuracy, Precision, Recall, and F1 Score), execution time, and model interpretability. Our findings reveal trade-offs between modularity, transparency, and latency, highlighting the need to align workflow architecture with deployment goals. The results provide practical insights for designing explainable, scalable, and resource-aware intrusion detection systems for real-time IoT security.