Détection automatisées des anomalies dans les infrastructures de Cloud Computing à l'aide des logs
Arthur Vervaet · HAL (Le Centre pour la Communication Scientifique Directe) · 2023
Cloud computing aims to optimize resource utilization while accommodating a large user base and elastic services. Within this context, cloud computing platforms bear the responsibility of managing their customers’ infrastructure. The management of an everexpanding number of IT resources poses a significant challenge. In this study, conducted in collaboration with 3DS OUTSCALE, a French public cloud provider, we investigate the potential of log data as a valuable source for automated anomaly detection within cloud computing platforms. Logs serve as a widely utilized information source for various purposes, including monitoring, diagnosing, performance evaluation, and maintenance. These logs are generated during runtime and provide insights into the current state of a system. However, achieving automated real-time anomaly detection based on log data remains a complex undertaking. The intricate nature of cloud computing platforms must be duly considered. Extracting relevant information from a multitude of logging sources and accounting for frequent code base evolution poses challenges and introduces the potential for errors. Furthermore, establishing log relationships within such systems is often an insurmountable task. Log parsing solutions aim to extract variables from the template of log messages. Our first contribution involves a comprehensive study of two state-of-the-art log parsing methods, investigating the impact of hyperparameter tuning and preprocessing on their accuracy. Given the laborious nature of labeling logs related to a cloud computing platform, we sought to identify potential generic values that enable accurate parsing across diverse scenarios. However, our research reveals the infeasibility of finding such requirements, thereby emphasizing the necessity for more robust parsing approaches. Our second contribution introduces USTEP, an innovative online log parsing approach that surpasses existing methods in terms of accuracy, efficiency, and robustness. Notably, USTEP achieves a constant worst-case parsing time complexity, distinguishing it from its predecessors for which the number of already detected templates is to be taken into account. Through a comparative analysis of five online log parsers using 13 open-source datasets and one derived from 3DS OUTSCALE systems, we demonstrate the superior performance of USTEP. Furthermore, we propose USTEP-UP, an architecture that enables the distributed execution of multiple USTEP instances. Our third contribution presents Monilog, a system architecture designed for automated log-based anomaly detection within log data streams. Monilog leverages model/metric pairs to predict log traffic patterns within a system and detect anomalies by identifying deviations in system behavior. Monilog forecasting models are powered by the recent advances in the deep learning field and is able to generate comprehensive reports that highlight the relevant system components and the associated applications. We implemented an instance of Monilog at cloud scale and conducted experimental analyses to evaluate its ability to forecast anomalous events, such as servers crashes resulting from virtualization issues. The results obtained strongly support our hypothesis regarding the utility of logs in detecting and predicting abnormal events. Our Monilog implementation successfully identified abnormal periods and provided valuable insights into the applications involved. With Monilog, we demonstrate the value of logs in predicting anomalies in such environments and provide a flexible architecture for future study. Our work on the parsing field with the proposal of USTEP and USTEP-UP not only provides us with additional information for building anomaly detection models but also has potential benefits for other log mining applications.