IT Equipment Monitoring and Analyzing System for Forecasting and Detecting Anomalies in Log Files Utilizing Machine Learning Techniques
Imam Yagoub, Muhammad Asif Khan, Jiyun Li · 2018
The ability to detect anomalies in application log files has attracted the attention of researchers over the past decade as it has become a challenging issue. Intuitively, a noticeable variation in the performance can be as a result of some natural causes (e.g., CPU workload variations and memory leaks) or from internal anomalies or errors that may cause performance failure or application crash. In here, an account of prediction and detection of the performance anomalies together with their causes has been reported. A framework for the detection of anomalies was particularly targeted onto the application log files whereby some quantity of historical data was acquired and analyzed. From the data set, a correlation analysis was demonstrated which data was then submitted for Machine-Learning (ML) forecasting and anomaly detection algorithms. The best algorithms were chosen based on accuracy and precision. In the second phase, CPU usage and memory utilization for the data points collected previously were analyzed. From the obtained results it was evident that combining the parameters for approximation aided by time-series models with ML forecasting and anomaly detection techniques provided excellent results as regards to prediction of performance anomalies. And the framework is robust enough to identify the applications causing these anomalies and abnormal behaviors.