Threshold based Technique to Detect Anomalies using Log Files
Toluwalope David Akande, Barjinder Kaur, Sajjad Dadkhah, Ali Akbar Ghorbani · 2022
Every action carried out on computer systems can be captured using log files. Proper scanning of log files can divulge security breaches. However, a large-scale data processing engine should analyze the log files due to the voluminous events in log files. This paper proposes an anomaly detection approach using a threshold to discriminate between regular and aberrant log files. The experiments are performed on HDFS, a publicly available log dataset. The system’s efficacy is evaluated using Robust Random Cut Forest (RRCF), an unsupervised tree-based approach where we achieved precision 97.10% and F1-score 98.47% results. Hadoop framework is utilized to run the experiments due to its capability of parallel processing tasks in less time, even on large datasets.