System log summarization via semi-Markov models of inter-arrival times
Erik M. Ferragut, Nicole Braden · 2011
A general recommendation for good cyber security is the diligent analysis of log data. However, logs are often very verbose and heterogeneous; their analysis presents an enor-mous burden on qualified cyber analysts. We propose an algorithm for automatically organizing cyber log data to fa-cilitate their efficient analysis. Our hypothesis is that events that occur with periodicity in the logs generally present fewer risks. These events may be results of synchronous com-puter operations (e.g., cron jobs or DHCP renewals). The organization of the data uses a probabilistic (hidden semi-Markov) model to combine these micro-events into macro-events, thereby making it easier to view their aggregate properties and to make decisions about security impact on the collection of events. We argue that considerable analyst effort will be saved with our approach.