ANALYSIS OF PENETRATION TESTING APPROACHES USING REINFORCEMENT LEARNING
Andrii Prytula, Leonid M. Kupershtein · Cybersecurity Education Science Technique · 2025
Penetration testing (PT) is an important method for ensuring digital security, which allows to assess the presence of vulnerabilities in systems and networks through attack simulations. Due to the rapid development of technologies and the growth of digital threats, there is a need to improve testing methods, in particular through the implementation of machine learning (ML) and reinforcement learning (RL) algorithms. The article discusses modern approaches to automating penetration testing using machine learning and reinforcement learning, which can significantly increase the efficiency and accuracy of the process. Penetration testing includes several stages, such as collecting information about the target system, scanning, analyzing threats and vulnerabilities, exploitation, generating a report, etc. Traditional methods often require significant human resources and time. The implementation of artificial intelligence (AI) and ML allows to automate these stages, which leads to a significant reduction in time and increased testing efficiency. In particular, the NLP-based approach demonstrates high potential for adapting to changes in the testing environment, allowing systems to independently improve their strategies over time, based on experience. The article reviews various approaches, including the use of deep learning and model-free NLP methods for penetration testing automation. The advantages and limitations of each approach are analyzed, including the importance of adaptability to environmental changes, high accuracy of vulnerability detection, and the difficulties that arise when integrating and configuring tools, especially for large and complex networks. Possible challenges associated with the use of significant computing power and the need to model specific conditions are also considered. As a result of the study, the most relevant approaches to penetration testing automation using reinforcement learning methods have been identified, which have significant potential for increasing the efficiency and adaptability of testing processes. Future research prospects focus on expanding the capabilities of NFP models for application in complex and large networks, as well as on integration with other cybersecurity platforms to create more comprehensive and efficient automated testing systems.