On Model Performance Estimation in Time Series Anomaly Detection
Arn Baudzus, Bin Li, Adnane Jadid, Emmanuel Müller · 2023
The usual way to quantify the performance of a novel algorithm in the field of classification, especially in time series anomaly detection, is to compare its performance against selected baseline competitors on selected data sets. There is a common sense in the community which data sets and baselines should be considered when evaluating the algorithm’s performance. Nevertheless, on which basis data sets and baselines should be selected is frequently discussed. In this paper, we propose an index for univariate time series data in anomaly detection based on information theory. The index shows an association with the AUC-score of an anomaly detection algorithm that is trained on the data, meaning that, the index can be used as a proxy for the “difficulty” for the classification task, this data set holds. A workflow to quantify this association using an interpretable classifier that relies on the index and a derived performance baseline is developed. The classifier performs within the margin of error of this performance baseline, meaning that we were not able to clearly mathematically show the association between index and AUC-score. We believe that our work, which unites mathematical concepts from information theory, physics, and computer science, is innovative and generally points in a promising direction that is worth investigating.