Design and Development of SSH Attack Detection Framework Using Reinforcement Learning Modules
Chirag Gupta, Shivam Bhavsar, Tanisha Nandwana, Rajendrane Rajmohan · 2025
In the dynamic realm of cloud computing, Distributed Denial-of-Service (DDoS) attacks targeting Secure Shell (SSH) protocols jeopardize the availability and reliability of web applications. Traditional detection methods often falter in distinguishing these malicious floods from legitimate traffic, exposing cloud systems to significant risks. This paper presents a novel SSH Attack Detection Framework that harnesses Qlearning, a reinforcement learning approach, to proactively detect and mitigate SSH-based DDoS attacks. Building on prior KClique clustering techniques, the framework integrates machine learning models-Random Forest and XGBoost-with realtime traffic analysis for initial threat detection, while a Q-learning agent adapts to evolving attack patterns. Deployed in a simulated AWS-like environment using TensorFlow, it analyzes synthetic datasets, capturing SSH logs and HTTP traffic to train a hybrid ML-RL model. The system achieves a 97 % detection accuracy and a false positive rate below 3%, outperforming tools like Snort and SVM. With automated mitigation strategies, such as enabling SYN cookies or throttling UDP traffic, it ensures rapid recovery-averaging 12 seconds during SYN flood simu-lations-while maintaining high service availability. This research underscores the power of reinforcement learning in delivering a scalable, resilient defense against evolving cyber threats in cloud infrastructures.