Revolutionizing Log Parsing for Modern Software Systems
Stefan Petrescu · 2023
With the tremendous log volume generated by modern systems, automated log analysis becomes indispensable for discovering crucial insights into the behavior of running systems. The foremost step subsumed in an automated log analysis pipeline, called log parsing, significantly influences its entire performance. Despite its significance, log parsing lacks quality implementations and, in practice, suffers from fundamental limitations, thereby creating a bottleneck for discovering valuable insights at the log line level. As a consequence, in my PhD thesis, I endeavor to explore a novel paradigm, called entity parsing, which goes beyond previous work by not only addressing the current limitations of log parsing but by exploring a new avenue for log parsing that could lead to significant advances in how we perform log analysis. As preliminary results show, entity parsing, despite solving a more difficult problem than conventional log parsing, is viable and obtains a significantly better accuracy on comparable datasets. By applying a first-principles approach, entity parsing is based on three fundamental components: data, machine learning, and source code. These components, leveraged within the framework of entity parsing, process logs in a way that could significantly improve the overall dependability of systems, enhancing their availability, security, and reliability.