An Iterative Clustering Approach for Tracking Server Logs for Monitoring SCADA EMS/DMS
Jit Biswas, David K. Y. Yau, Ming Yu, Kon Ih Lunn, Zihao Li, Chua Kian Wee, Tan Keng Nan, Zhang Zhimin, Jimmy Chua Wai Hong, Tso Wai Ann, Heng Yong Kean · 2021
Due to continuity of operation, software upgrades and patches may need to be incorporated without adequate testing. Tracking system log messages allows us to compare a system’s behavior over periods of time. We present an iterative clustering approach for weekly system logs that are maintained by servers in SCADA EMS/DMS systems. The main goal is to monitor behavior from one week to the next, for the purpose of summary reporting and diagnostics. The algorithm identifies clusters iteratively by reducing the data-set to a remaining set of outliers at each iteration. Details of the identified clusters are retained for further exploration and analytics. A set of tracking scores is obtained as a concise representation of weekly system behavior. The tracking scores are based on different metrics. We demonstrate the application of our algorithms using two real-life datasets from SCADA EMS/DMS. We also extend basic clustering algorithm through application of template matching rules to assign feature labels to each line of an input log, thereby preparing the datasets for machine learning applications. The algorithms presented are proposed as a means for monitoring weekly operations as well as studying the effect of long term changes caused by patched and upgraded software.11This research was funded by a grant from the Energy Market Authority of Singapore (EMA) through Grant RGEMA1901, under the Energy Programme National Cybersecurity R&D Grant call 2018 (NRF2018-NCR003)