CANARI: A Monitoring Framework for Cluster Analysis and Node Assessment for Resource Integrity
Ryan T DeRue, Jacob C. Verburgt · 2024
In providing High Performance Computing (HPC) systems to research faculty at universities, research computing facilitators must often strike a balance between providing the most up-to-date versions of commonly used software packages and libraries while also ensuring that the software ecosystem on the cluster is stable enough that version changes do not cause performance degradation to existing workflows. Additionally, the modern data centers where these ecosystems are running are very large, intricately complex systems that provide many points of failure. The sum of these two challenges present the need for tools that can help to ensure that these systems, and the software running on those systems, are continuing to perform at their expected levels. To this end, this paper will present a framework for Cluster Analysis and Node Assessment for Resource Integrity that we call CANARI. CANARI was developed and used at the Rosen Center for Advanced Computing (RCAC) to continuously monitor the availability of nodes in our clusters as well as their performance against synthetic benchmarks, ingest that performance data into a persistent database, mark nodes displaying performance regression offline, and provide summary reports and real-time updates to the Slack instance used at RCAC by using Slack’s API.