Ensemble Deep Learning based Real-time Log Anomaly Detection
Abhiram Puranik, Akanksha V Akkihal, R K Suhas, Y P Dhanush Patel, H. B. Mahesh · 2023
Applications/Services generate a large amount of logs constantly. It is a crucial task for administrators to extract useful information from logs. Analysis of these logs is essential to find anomalies and fix issues quickly in the application/service. It is a very tedious task to go through all the log messages manually to find anomalies. This points to the need for an anomaly detection system. An anomaly detection system automates the analysis of logs generated by the system to discover abnormalities in the system execution. To build the anomaly detection system, three deep learning models i.e. LSTM (Long short-term memory), BERT (Bidirectional Encoder Representations from Transformers), and CNN (Convolutional neural network) is proposed. Ensemble of these three models using the voting technique is built. The ensemble of deep learning models is done to arrive at an optimal prediction for anomalies as these models compensate for the drawbacks of each other. This paper also talks about the implementation of real-time anomaly detection by leveraging the existing bulk log parser called Drain to do online parsing. A user interface is provided to view the detected anomalies and filter the logs accordingly.