Anomaly Detection for ETL Packages Runtime: A Machine Learning Approach
Behiye Alak, Alp Revanbahş, Nilay Argün, Ayşegül Şenol Çalım · 2023
In today's dynamic business environment., financial institutions and banks use ETL (Extract, Transform, Load) packages to perform data integration and optimize business processes. However, sudden changes and anomalies in the runtimes of these packages can negatively affect the workflow and cause delays and customer dissatisfaction. In this study, it is aimed to improve the service quality of enterprises, to provide operational efficiency, and to better manage business processes with the detection of anomalies in the runtime of ETL packages. In this project, the Interquartile Range (IQR) method for outlier detection and data labeling, and K-Nearest Neighbors (KNN) and eXtreme Gradient Boosting (XGBoost) algorithms for machine learning are used. The dataset includes approximately 2 thousand ETL packages runtime data in the last years. The results show that KNN and XGBoost models provide a weighted-averaged F1 score of 99.9%. This study provides significant benefits in areas such as customer satisfaction and cost reduction. Future works may address subjects such as using larger and more feature-rich datasets and different anomaly detection methods.