Agentic AI for Cyber Deception: A Gestalt Game-Theoretic Approach to Defending Against Botnet DDoS Attacks
Quanyan Zhu, Muhammad Akram Al Bari · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · 2026
Agentic AI represents a transformative approach to cybersecurity, offering modular, flexible, and intelligent architectures capable of orchestrating complex defense operations. A central challenge lies in the design and optimization of such systems; specifically, how to construct an optimal workflow and configure agents to effectively accomplish a mission objective. This paper introduces a principled framework for modeling and optimizing agentic AI workflows, with a specific application to cyber deception. We formalize agent coordination using an extensive-form representation that captures inter-agent dependencies, enabling recursive utility estimation and workflow-level optimization via dynamic programming. At the core of our approach are large language model (LLM)-driven agents that manage tasks such as honeypot deployment, adaptive engagement, and semantic analysis of attacker behavior. Through a detailed honeynet case study, we compare a utility-aware adaptive strategy against a static, greedy baseline under operational budget constraints. The results of 500 simulated epochs show that the adaptive workflow yields a 2.5-fold increase in successful deception episodes and reduces cost per success by more than 58%, while maintaining greater stability across attacker types. These findings highlight the potential of designing optimal agentic AI to enable resource-aware and resilient cyber defense systems.