Statistical Anomaly Detection on Metadata Streams via Commodity Software to Protect Company Infrastructure: A Case Study
Christine Chen, James Gurganus · 2017
As a company grows, its infrastructure naturally must grow to support it. The resulting mountains of infrastructure metadata contain valuable information on the health and wellbeing of a company's systems. For example, an abnormally low disk write rate to a file server may indicate that a regularly scheduled task has failed to start. The hypothesis of this case study is that such metadata streams can be effectively utilized by implementing statistical anomaly detection methods via commodity software (Splunk, in this case). These methods were tested on server metadata in a ransomware simulation and on server metadata from active file and production servers. In the ransomware simulation, the alerting system detected the ransomware behavior five minutes after an encryption event began in the simulation environment and alerted steadily for the duration of the simulation. In the week-long experiment over 11 file servers and production servers, a total of 1,484 alerts were generated. Applying simple correlation techniques created a more concentrated information stream with 77 events. These results confirm the value of metadata in identifying system anomalies and providing another layer of defense against malicious threats. The relatively simple anomaly detection techniques highlight the increasing practicality of behavioral analytics-it can only be a matter of time before such techniques will be ubiquitous.