Building Observability into Full-Stack Systems: Metrics That Matter
Kiran Kumar Pappula, Sunil Anasuri, Guru Pramod Rusum · International Journal of Emerging Research in Engineering and Technology · 2021
In the paper, a framework of observability in full-stack systems is defined. It links frontend performance and backend health metrics, log aggregation and traceability. The art (or science) of observability is shifting towards data-rich, event-driven observability that is an important step towards resilient, scalable systems. The full-stack paradigm requires the telemetry to be integrated at the frontend, backend, infrastructure, and application levels. We propose a unified model that quantifies the relationship between the behaviours of systems and the experiences of users with structured metrics, logs and traces. Our framework utilizes the open standards OpenTelemetry and integrates the distributed tracing tools like Jaeger, Prometheus, in order to collect metrics, and the ELK stack to aggregate the logs. The objective is to have insight into profound levels of system state and performance bottlenecks, as well as anomaly detection. The architecture is organized in the form of five strata- Instrumentation, Telemetry Collection, Analysis, Visualization, and Action. Each of the levels is correlated with technical elements and levels of observability. An analytic model is likewise formulated to measure observability coverage in terms of signal density and correlation coefficient of traces and metrics. The framework was evaluated through a case study of an e-commerce application based on microservices and a frontend interface using React.js. Mean Time to Detect (MTTD) and Mean Time To Resolve (MTTR) showed great improvements in performance. We also mention telemetry noise, data storage cost and cross-domain correlation as the challenges in this case. Our results give a viable route that all organizations seeking to implement observability in production can follow