Monitoring and scaling GPU workloads in production with Nvidia DCGM and Prometheus
Srinivas Kola · ISCSITR-INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING · 2024
The paper introduces a comprehensive framework for monitoring and dynamically scaling GPU workloads within production-grade computing environments, leveraging the capabilities of the Nvidia Data Center GPU Manager (DCGM) in conjunction with the Prometheus monitoring and alerting toolkit.The proposed architecture is designed to provide granular visibility into GPU performance by collecting a diverse range of telemetry data, including core utilization rates, memory allocation and consumption patterns, power draw, and thermal metrics.This fine-grained data capture enables detailed operational insight, allowing for both reactive and proactive system management.The collected telemetry is systematically processed and stored in a high-performance timeseries database, facilitating continuous, low-latency analysis of workload behavior over time.This design choice ensures that monitoring remains scalable and responsive under production workloads, while supporting sophisticated querying and visualization for operational decision-making.Integration with modern orchestration frameworksparticularly Kubernetes-allows the system to feed live telemetry into automated scaling logic, thereby enabling the cluster to adjust GPU resource allocation in near real time based on evolving workload demands.This closed-loop feedback mechanism ensures that the system can maintain optimal utilization levels, mitigate performance bottlenecks, and uphold service-level objectives without manual intervention.The experimental evaluation, conducted within a Kubernetes-managed GPU cluster, demonstrates that the proposed monitoring and scaling strategy consistently sustains high GPU utilization while effectively responding to operational anomalies and threshold-based alerts.The results highlight its ability to detect and address potential performance degradations before they impact end users.The study also provides valuable implementation-level insights for practitioners, outlining practical considerations such as network overhead, monitoring granularity trade-offs, and the tuning of alerting thresholds to balance responsiveness with stability.The authors acknowledge certain limitations inherent in the current system, including constraints in predictive capability and hardware compatibility.The system's scaling logic is presently reactive, relying on predefined thresholds, and does not yet incorporate predictive models capable of anticipating future demand surges.Furthermore, its hardware coverage is largely limited to Nvidia GPUs, reducing applicability in heterogeneous computing environments.The paper concludes by outlining future work aimed at addressing these gaps, such as incorporating predictive scaling algorithms, extending hardware support beyond Nvidia platforms, and exploring integration with advanced scheduling and workload placement strategies to further optimize GPU utilization in large-scale deployments.