Automated Event Identification from System Logs Using Natural Language Processing
Abhishek Dwaraki, Shachi Kumary, Tilman Wolf · 2020 International Conference on Computing, Networking and Communications (ICNC) · 2020
Legacy methods of troubleshooting and root-cause analysis in networks involve time-consuming manual analysis of log files to understand and debug faults. Logs track system states and ensure that critical events are recorded to help troubleshooting and root cause analysis. Log files and their corresponding entries are ubiquitous and are important sources of information for interpreting system state. As methods have improved, many automated log collection and reporting systems have been developed to ease the burden on system administrators. With the increased pervasiveness of machine learning, this technology is capable of changing the way newer network management systems are built. We propose an event identification and management framework based on natural language processing concepts. Our system processes events in a log file as a natural language sequence and builds models of the extracted events to be used in various online or post-processing scenarios. We demonstrate how text categorization and sentence similarity concepts can be used to automatically identify events in logs. We also illustrate the advantages of our event extraction framework in different use cases and how our system helps to make network troubleshooting and management more efficient.