Anomaly Detection for Log using AutoM and Auto-Sklearn Classification Algorithm
G. M. Karthik, Ankur Gupta, Sh. Rajesh Gupta, Amaresh Jha, A. Sivasangari, Bhabani Prasad Mishra · 2024
In the realm of application development, logs represent a pivotal resource for the identification and comprehension of application performance, as well as the detection of anomalies, all of which are essential for ensuring the overall efficiency of an application. The proliferation of users has resulted in a substantial increase in the volume of logs available for analysis in the field of log analysis research. Typically, these logs exhibit an unstructured format, largely dictated by the developer’s design choices. Consequently, there is a need for log parsing techniques to in still structure into these logs, enabling the application of machine learning algorithms for the prediction of anomalies. In the present research, the primary objective is the prediction of anomalies within Hadoop Distributed File System (HDFS) logs utilizing an AutoML (Automated Machine Learning) tool known as Auto-Sklearn. The research outcomes illustrate the effectiveness of the proposed model, which attains an impressive accuracy rate of $\mathbf{9 9 \%}$ when applied to the dataset. This study underscores the critical role of logs in application development, highlights the importance of log structure transformation, and demonstrates the substantial predictive capabilities of the Auto-Sklearn model in the context of HDFS log analysis.