Application of Multi-Level Log Management and Analysis System in Large-Scale Microservice Architectures
Bo Li, Zhenghao Qian, Mingdong He, Xiaojuan Zheng, Xuewu Li · 2025
With the widespread adoption of microservice architectures in distributed systems, the efficient management and analysis of log data has become a significant challenge. This paper presents a multi-level log management and analysis system designed to address the log processing challenges in large-scale microservice environments. The system enhances log management efficiency through strategies such as hierarchical storage, compression optimization, stream transmission, and real-time analysis. It uses the Ceph distributed storage system and the Zstandard compression algorithm to optimize storage performance, ensuring fast data access in high-concurrency environments. By integrating Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers (BRET) models, the system implements an intelligent earlywarning mechanism based on log data, enabling real-time detection and diagnosis of system faults. Experimental results show that the system significantly outperforms traditional log management systems in throughput, query response time, and fault recovery, and can operate stably in large-scale microservice architectures. The multi-level log management and analysis system proposed in this paper provides an effective solution for log management in modern microservice architectures.