Enhancing Packet Tracing of Microservices in Container Overlay Networks using eBPF
Chunghan Lee, Reina Yoshitani, Toshio Hirotsu · 2022
The microservices architecture has been rapidly adopted to latency-sensitive applications. The architecture of these applications and the container overlay networks on servers are also complex. Distributed tracing is widely adopted in microservice applications; it suffers from the drawback of only focusing on service discovery and latency-based monitoring at an application layer, making it still challenging to monitor container overlay networks using distributed tracing. In this paper, we present an extended Berkeley Packet Filter (eBPF)-based packet tracing method using distributed tracing for latency measurement in the container overlay network. To efficiently detect the trace context on a HTTP payload, we moved the trace context position just behind the HTTP request line using sidecar proxy. Our tracing method gathered the HTTP packets that had the trace context and measured the latency using eBPF. Our evaluation was conducted using open-source benchmarks on Kubernetes; the results showed that the proposed tracing header format reduced the HTTP payload search space by up to approximately 80, and there was no significant change in end-to-end latency. Moreover, our eBPF-based tracing method presented similar latency characteristics on the overlay network in comparison with the characteristics of the packet-level traces obtained under tcpdump.