Multi-Layer Observability for Fault Localization in Microservices Based Systems
Rupashree Rangaiyengar, Raghavan Komondoor, Raveendra Kumar Medicherla · 2023
For cloud native microservice monitoring and incident detection, companies and developers tend to largely focus only on generating logs, metrics, and traces at the application layer. However, in order to enable precise fault localization, it is necessary to access and correlate logs pertaining to a single end-user request across non-application layers as well, such as the load balancer at the front and the database at the back end. In this paper, we propose an observability library and an observability platform that addresses this problem and generates alerts that precisely point to fault locations. Logs at multiple layers are tagged with a common request identifier that helps in performing correlation. The observability platform is architected such that it lends itself to extensions to catch multiple types of errors and issues. The proposed observability platform has been tested on five open source benchmarks. The results confirm that our tool can be used deterministically and precisely to detect elusive issues.