Toward an Automated Real-Time Anomaly Detection Engine in Microservice Architectures
Md Khan · 2023
With the growing demand for micro-service applications and cloud-based software, service providers like Amazon, Netflix, and eBay, are finding it difficult to detect the failures in their increasing number of interlinked services of the applications.Various machine-learning techniques have been proposed to detect these failures.However, one of the significant challenges anomaly detectors face is model performance decay over time due to data pattern changes in time series data [1].Another major issue is detecting a high number of false positives( false anomalies) [2], creating false alerts that question the credibility of the detector's performance.This thesis attempts to tackle these issues mentioned above by building a dynamic automated pipeline for an unsupervised anomaly detection engine for a micro-services application to predict anomalies in the system's behaviour.The anomaly detection engine consists of five modules: Data Collection and Pre-processing Module, Forecaster Training Module, Detector Module, Fault Injection Module, and Sending Alert Module.The Data collection and Pre-processing Module collects the data through Prometheus.The Forecaster Training Module trains the forecaster in an online environment, updating the model's parameters and training reconstruction error.The detector module predicts anomalies on new forecasted data based on the threshold calculated using training reconstruction error, and Sending Alert Module sends the reported anomalies to the application's operator through the mail.We evaluated a deep-learning approach with an LSTM forecaster and a statistical approach with SARIMAX as a forecaster.A Fault Injection Module is added to the system to evaluate the model that injects simulated anomalies.We evaluated and compared the performance of the proposed model trained in a dynamic environment with the baseline model trained in a non-dynamic environment.The proposed model outperforms the baseline model forecaster, which has low average iii MAE and MAPE values on validation sample, and the detector has higher precision and recall values than the baseline model's detector.LSTM and SARIMAX models perform similarly in short-term forecasting, while LSTM outperforms SARIMAX in long-term forecasting, having a low false positive rate.In terms of computational costs, LSTM is three times faster than the statistical approach.This proposed model helps service providers like Netflix and Amazon reduce the rate of predicting false alarms in the application and reduce the performance decay rate avoiding violation of the Service Level Agreement between them and their clients.iv I dedicated this thesis to my God for granting me the strength to finish my work with my recent ailing health problems.Secondly, I want to dedicate this work to my parents.