Server-Language Processing: A Semi-Supervised approach to Server Failure Detection
Sonali Syngal, Sangam Verma, Kandukuri Karthik, Yatin Katyal, Soumyadeep Ghosh · 2021
As industrial systems continue to grow in terms of scale and complexity, having an effective as well as proactive failure management approach helps mitigate the impact of server failure. While supervised methods fail to perform well in real-world servers due to label noise in log data as well as their failure to detect unseen failures, unsupervised techniques are often too naive to differentiate between complex log structures. We propose a NLP based semi-supervised solution that learns the complex understanding of healthy and failure log patterns using an ensemble of deep learning based density and sequential solutions. Our hypothesis is that server logs follow a language of their own, which we attempt to decipher through Server-Language Processing. Experimental evaluations on real world log data show that our proposed solution outperforms other existing log-based anomaly detection methods for real world application. The solution was implemented for 3000 servers for 6 months of log data, and was able to pick up server failures upto 2 weeks in advance without raising an excess of false alarms.