Tracemesh: Scalable and Streaming Sampling for Distributed Traces
Zhuangbin Chen, Zhihan Jiang, Yuxin Su, Michael Rung-Tsong Lyu, Zibin Zheng · 2024
Distributed tracing serves as a fundamental element in the monitoring of cloud-based and datacenter systems. It provides visibility into the full life cycle of a request or operation across multiple services, which is essential for understanding system dependencies and performance bottlenecks. To mitigate computational and storage overheads, most tracing frameworks adopt a uniform sampling strategy, which inevitably captures overlapping and redundant information. More advanced methods employ learning-based approaches to bias the sampling toward more informative traces. However, existing methods fall short of considering the high-dimensional and dynamic nature of trace data, which is essential for the production deployment of trace sampling. To address these practical challenges, in this paper we present Trace Mesh,a scalable and streaming sampler for distributed traces. Tracemesh employs Locality-Sensitivity Hashing (LSH) to improve sampling efficiency by projecting traces into a low-dimensional space while preserving their similarity. In this process, Tracemesh accommodates previously unseen trace features in a unified and streamlined way. Subsequently, Tracemesh samples traces through evolving clustering, which dynamically adjusts the sampling decision to avoid over-sampling of recurring traces. The proposed method is evaluated with trace data collected from both open-source microservice benchmarks and production service systems. Ex-perimental results demonstrate that Tracemesh outperforms state-of-the-art methods by a significant margin in both sampling accuracy and efficiency.