Low cost log statistical filtering in the real world chained tokens and Bayesian filters
Russel W. Havens, Chia-Chi Teng · 2011
Log files are valuable resources for systems administrators to trouble shoot and prevent failures. Bayesian filters can be utilized as effective log entry filters, reducing the log noise that often makes today's huge log files daunting. The effectiveness of three off-the-shelf, open source Bayesian spam filters, SpamAssassin, SpamBayes and Bogofilter, is tested for differentiating syslog entries from a corpus taken from production Linux servers at the School of Technology, Brigham Young University. Additionally, the effectiveness of word chaining, stacked word chaining and the normalization of numbers is discussed and quantified. The effectiveness of two of the filters is improved significantly by chaining words in two or three word chains, particularly for SpamAssassin, which saw a significant improvement in differentiation for the log corpus. Results of preliminary experiments have confirmed that these filters can effectively process typical systems logs and have potential applications in the future.