An optimized reinforcement learning based MTD mutation strategy for securing edge IoT against DDoS attack
Amir Javadpour, Forough Ja’fari, Chafika Benzaïd, Tarik Taleb · Journal of Information Security and Applications · 2025
Distributed Denial of Service (DDoS) attacks are among the most destructive and challenging threats to mitigate for computer networks, particularly in edge IoT environments. Moving Target Defense (MTD) is a promising security mechanism that undermines the adversary’s gathered information by dynamically altering the attack surface. A selection of network nodes is chosen for mutation, and these changes hinder the adversary from achieving their objectives. However, identifying the optimal set of nodes for effectively and efficiently mitigating a DDoS attack remains a significant challenge. Existing MTD approaches have only considered a single factor—either the node’s vulnerability level or connectivity—and often lack generality and scalability for real-world IoT implementations. In this paper, we propose an enhanced MTD approach called CVbMA (Connection- and Vulnerability-based MTD Approach) that jointly considers both the vulnerability levels and connection weights of nodes to inform mutation strategies. To ensure practical applicability and adaptability, we develop a cost-aware Reinforcement Learning (RL) framework that incorporates explicit mutation costs into the reward function and utilizes neural ranking and model compression for scalability. Extensive evaluations are conducted using both Mininet-based simulations and a physical IoT testbed with real attack traces and heterogeneous devices. Comprehensive benchmarking and ablation studies against state-of-the-art MTD baselines demonstrate that the proposed framework significantly reduces the adversary’s success rate and incidents of server crashes, while maintaining low overhead and achieving high adaptivity. A detailed analysis of real-world deployments highlights the robustness of systems under operational constraints, including fluctuating latency, hardware diversity, and asynchronous events. Limitations and future enhancements, including topology-aware RL, adaptive mutation scheduling, and continuous model updates, are discussed. The results affirm the practical, scalable, and robust potential of cost-sensitive RL-based MTD for next-generation IoT security.