AI in Penetration Testing: A Systematic Mapping Study
Sulaiman O Alwabisi · 2025
The integration of Artificial Intelligence (AI) into penetration testing presents transformative opportunities for enhancing cybersecurity practices. This systematic mapping study investigates the current state of AI-driven penetration testing by reviewing 57 primary studies published between 2015 and 2025. Through a rigorous quality assessment process based on ten criteria, high-quality papers were selected for detailed analysis. The study is guided by four research questions: (RQ1) What are the key AI techniques currently utilized in penetration testing? (RQ2) What are the major challenges and limitations associated with integrating AI into penetration testing? (RQ3) How effective are AI-driven techniques in enhancing the penetration testing process? (RQ4) What are the current research trends, gaps, and future directions? Findings reveal that reinforcement learning (RL), deep learning (DL), generative AI models, and supervised learning are the most frequently adopted techniques. These approaches contribute significantly to automation, improved vulnerability detection, scalability, and optimization of penetration testing workflows. However, several challenges persist, including scalability limitations, training inefficiencies, integration difficulties, and ethical concerns related to AI bias and misuse. Emerging trends indicate growing interest in large language models (LLMs) and hierarchical reinforcement learning to address current limitations. This study highlights the evolving landscape of AI-assisted penetration testing and provides a roadmap for future research aimed at enhancing model effectiveness, real-world applicability, and ethical deployment. The insights gathered from this mapping study offer valuable guidance for researchers and practitioners seeking to advance automated and intelligent cybersecurity solutions.