A Unified Framework for Robust Encrypted Malicious Traffic Detection in Adverse Environments via Graph Structure Learning
Jianjin Zhao, Zhiwei Cui, Junsong Fu, Meng Shen, Qi Li · IEEE Transactions on Network Science and Engineering · 2025
The widespread adoption of encryption protocols enables attackers to conceal malicious activities within encrypted traffic, rendering traditional detection methods ineffective. Graph Neural Networks (GNNs) have emerged as a promising solution by modeling network objects and their interactions within graph representations to capture the collaborative behavioral patterns of complex threat activities. However, the well-performed premise of GNNs does not always hold in adverse environments, leading to unsatisfactory performance, suffering from three critical issues including (1) incomplete information analysis, where heterogeneous relations among network objects are often overlooked (2) lack of solutions for evasion techniques, as existing methods focus on robust representation learning but fail to correct adversarial distortions, and (3) limited robustness evaluation, relying on synthetic feature perturbations rather than raw traffic manipulations in line with real-world attacks. To address these issues, we proposeRETA, a unified framework for robust encrypted malicious traffic detection via graph structure learning. First,RETAunifies heterogeneous subgraphs capturing semantic metapaths and homogeneous subgraphs modeling behavioral similarities among encrypted sessions and takes a tailored Heterogeneous Graph Attention Network (HAN) encoder for neighborhood information aggregation. Then, it employs a unified graph structure learning framework to correct noisy relations induced by evasion techniques through channel attention-based aggregation and Bayesian inference-based estimation. Following an iterative manner,RETAmutually improves relation modeling and detection robustness. Finally,RETAsimulates various realistic adverse conditions by modifying raw traffic captures, ensuring comprehensive robustness evaluations against network fluctuations and adversarial attacks. Extensive experiments demonstrate the superior robustness ofRETA, significantly improving detection performance in adverse environments. Even under extreme adverse conditions (i.e., 30% packet loss rate and 5 perturbation edges),RETAstill shows significant advantages, delivering 8.94% and 4.85% accuracy improvements over the baseline models on average.