LogTIW:A log anomaly detection model based on TF-IDF weighted semantic features
Jia Kang, Junfeng Zhao, Zhengxin Li · 2024
In online computer systems, the detection of anomalous events is crucial for protecting the system from failures. System logs record detailed information about computing events and are widely used for system state analysis. Existing log-based anomaly detection methods are affected by the quality of semantic vectors. Semantic vectors obtained using Word2Vec or BERT only represent the semantics of sentences, disregarding the importance of individual semantics. To address these limitations, we have designed a weighted approach based on TF-IDF. Unlike LogRobust, which applies weighting during the construction of semantic features, LogTIW separately constructs semantic features and TF-IDF features of log templates after parsing. We extract semantic features using Transformer and LSTM models, and extract weighted features carried by TF-IDF using LSTM and linear layers. Then, the semantic features are weighted using the extracted weight features. Experimental results demonstrate that on the publicly available HDFS log dataset, LogTIW achieves precision, recall, and F1 scores all exceeding 99%. LogTIW outperforms state-of-the-art methods in anomaly detection.