Design and Development of a Log Management System Based on Cloud Native Architecture
Yuchen Sun, Yanpiao Chen, Haotian Zhao, Shan Peng · 2023
With the rapidly development and wide application of cloud computing and service computing, the use of cloud native architecture has become a mainstream trend in software development and deployment. In this booming background, logs have become the key information for each application to record its operation process throughout its life cycle, while existing log management systems are usually designed for specific production environments, which are not able to effectively meet the growing and diversified scenarios. The existing log management systems are usually designed for specific production environments and cannot effectively meet the needs of the growing number of diverse scenarios. Therefore, how to effectively handle log information from multiple sources becomes a key issue. In this paper, we design and develop a log management platform based on cloud-native architecture, which provides log collection, transmission, storage, and system management functions, and at the same time can be oriented to the cloud-native environment and other third-party applications for collection, providing a powerful tool for developers and operation and maintenance personnel. In addition, we designed an anomaly detection system using large language models for log parsing and used this system for anomaly detection in platform logs. Finally, the effectiveness of the system was verified by commonly used log data. The log management platform based on cloud-native architecture passed the test and has been utilized in our real production.