HTA-BotDef: A novel hierarchical tree aggregation for scalable and privacy preserving botnet attack detection with federated learning
Md. Alamgir Hossain, Md. Samiul Islam · Journal of Information and Intelligence · 2025
The accelerating sophistication and diversity of botnet attacks continue to undermine network resilience, necessitating defense frameworks that are both decentralized and privacy-preserving. This study presents HTA-BotDef, a novel Hierarchical Tree Aggregation (HTA) framework designed for scalable and secure botnet detection through Federated Learning (FL). Unlike conventional centralized paradigms, HTA-BotDef enables distributed clients to collaboratively train a global detection model without exchanging raw data, thereby preserving privacy while maintaining high detection efficacy. To emulate real-world, non-IID network conditions, the framework integrates heterogeneous datasets including N-BaIoT, ISCX, MedBIoT, and NCC2, encompassing diverse botnet families such as Mirai, Bashlite, IRC Bot, Torii, Neris, and RboT. The proposed architecture combines advanced feature selection strategies, correlation analysis, mutual information, and Principal Component Analysis (PCA) with Random Forest-based local classifiers, aggregated via the hierarchical tree mechanism to form an adaptive and resilient global model. Dynamic hyperparameter optimization further enhances cross-client generalization and convergence stability. Extensive experiments demonstrate consistent and near-perfect performance, with accuracy, precision, and recall values exceeding 99.99% even under severe class imbalance. The findings establish HTA-BotDef as a highly scalable and privacy-preserving solution for real-time botnet detection, offering a significant advancement toward trustworthy and decentralized network defense systems.