Next-Generation Observability for Spark Workload Anomaly Detection: Integrating eBPF, Distributed Tracing, and SLO-Based Control Loops
Jonathan Benson · Zenodo (CERN European Organization for Nuclear Research) · 2026
Abstract Detecting anomalies in enterprise Apache Spark workloads increasingly requires visibility that neither traditional application-level logging nor coarse infrastructure metrics can provide: kernel-level insight into what a Spark executor is actually doing on the machine, correlated across the distributed stages of a job, and connected to an automated response mechanism grounded in service-level objectives (SLOs). This paper develops a three-layer observability framework for Spark workload anomaly detection, integrating extended Berkeley Packet Filter (eBPF)-based kernel telemetry, OpenTelemetry-standard distributed tracing with cross-modal anomaly detection, and an SLO-based closed-loop control mechanism, building on Mohammed's (2026) SLO-driven reliability framework for cloud-native data platforms. Synthesizing current research and industry practice on eBPF-based zero-code instrumentation, distributed-tracing anomaly-detection benchmarks (including multimodal microservice datasets and industrial-grade time-series benchmarks), and control-theoretic formulations of SLO enforcement and error budgets, the paper proposes a three-layer architecture, Kernel-Level Telemetry Collection, Distributed Trace Correlation & Anomaly Detection, and SLO-Based Control Loop and situates each layer within documented technique and reported evidence. The framework is illustrated through disclosed, non-empirical proof-of-concept scenarios and a control-loop diagram grounded in a simple proportional-control formulation of SLO enforcement. The analysis does not report primary experimental or production data beyond what is cited from public sources and the exact arithmetic of error-budget calculation, which is independently verifiable. The paper discusses theoretical, engineering, and organizational implications for enterprise Spark observability practice and proposes a structured agenda for empirical validation.