AI-Driven System for Automated Anomaly Detection in Cloud Through Continuous Monitoring of Logs, Metrics, and Performance Data
Meenakshi Bansal · Cuestiones de Fisioterapia · 2024
The increasing complexity of cloud computing environments necessitates robust, automated monitoring systems to ensure high availability and operational efficiency. Traditional manual anomaly detection methods are no longer sufficient due to their limited scalability, high error rates, and delayed response times. This research proposes a machine learning-based anomaly detection system designed to proactively monitor cloud operations by analyzing real-time streams of logs, system metrics, and performance indicators. The system ingests diverse data sources including timestamped logs, CPU and memory utilization, network traffic, response times, and error rates, each tagged with unique resource identifiers. It leverages both labeled and unlabelled datasets for comprehensive model training and evaluation. A hybrid approach is adopted using supervised algorithms—Support Vector Machine (SVM), Random Forest, Deep Neural Network (DNN), and Extreme Gradient Boosting (XGBoost). SVM achieves 97% accuracy on labeled historical data, while DNN reaches 99.2% by modeling complex nonlinear patterns, and XGBoost achieves 98.8% by optimizing performance through gradient boosting on decision trees. Notably, Random Forest attains 100% accuracy across both labeled and unlabelled scenarios, demonstrating exceptional generalization through ensemble learning. Detected anomalies are classified by severity and type, with each assigned a confidence score to guide timely responses—either automated or manual. This intelligent framework substantially enhances anomaly detection accuracy, minimizes false positives, and supports resilient cloud operations. Its modular, scalable architecture ensures compatibility with existing cloud infrastructures. Future enhancements will focus on real-time adaptability, cross-cloud deployment, and integration with predictive analytics and self-healing protocols to enable autonomous cloud management.