Process resource usage anomaly detection method based on process call stack in Linux Server
Bing Liu, Ruilian Xie, Lijun Chen · 2023
The call stack is a trace of function calls when a program is running. By analyzing the call stack, we can learn the execution path of the program, the relationship between functions and the number of function calls, which is of great significance for program optimization and troubleshooting. However, the call stack information is often very large, especially in complex application scenarios, the analysis and processing efficiency is very low, or even impossible to complete. Therefore, efficient statistics and analysis tools for the call stack are particularly important. This paper proposes an resource usage anomaly detection scheme for process behavior based on call stack coding and anomaly detection algorithm. It obtains the call stack by eBPF technology and establishes a hash table by map mechanism. It samples the call stack codes of all processes running in the machine at a specific frequency during the detection, and records the number of samples and execution time of each different call stack code. A user mode program reads data from the buffer. The call stack statistics is analyzed in user space, and the abnormal number of process call stacks is found by anomaly detection algorithm every period of time, which is convenient to find the call bottleneck of the machine causing high load. The abnormal call stacks are used to optimize the program and improve the performance of the machine, and are compatible with the data collected by Perf. The tool is efficient, accurate, scalable, and can quickly locate performance problems in the production environment.