Distributed Tracing for Cascading Changes of Objects in the Kubernetes Control Plane
Tomoyuki Ehira, Daisuke Kotani, Yasuo Okabe · 2025
Kubernetes is a container orchestration system that employs a declarative configuration management approach. In Kubernetes, each desired and actual state is represented in an "object", and multiple controllers autonomously monitor related objects and update their objects towards the desired state in the control plane. Because of this design, changes to one object cause changes to other objects in a chain, and the time to complete these cascading changes is an important metric for cluster operators to maintain application service quality on the cluster. However, there is no practical way to observe this kind of cascading change in detail, such as sequence of objects updated and the elapsed time of this update. Distributed tracing techniques, which are used in the microservice architecture to monitor application performance, are mainly based on RPC model that clearly identify request and response pairs, thus they are not directly applicable to the control plane of Kubernetes, where controllers just monitor objects and autonomously trigger changes to objects. In this paper, we propose a system that automatically traces changes to objects in the control plane. Our method adds one identifier, a Change Propagation ID (CPID), to the metadata of each object, and the controller that observes an object change propagates CPID in the object to the objects that the controller updates. When multiple changes need to be merged on an object, a new CPID is generated, and a pair of the original CPIDs and the new CPID is sent to the trace server for recording and analysis. Through evaluation, we confirmed that the proposed system can capture change propagation and its time, and the overhead of the proposed system is not significant.