SYMBIOMON: A High-Performance, Composable Monitoring Service
Srinivasan Ramesh, Robert B. Ross, Matthieu Dorier, Allen D. Malony, Philip Carns, Kevin Huck · 2021
High-performance computing (HPC) software is evolving to support an increasingly diverse set of applications and heterogeneous hardware architectures. As part of this evolution, the construction of scientific software has shifted from a traditional monolithic message passing interface executable model to a coupled, services-style model in which simulations run alongside a host of distributed HPC data services within the same batch job allocation. Microservices have emerged as a powerful new way to build these distributed data services through a composition model. However, performance analysis of composed microservices is a daunting challenge. It requires collecting, monitoring, aggre-gating, and exporting performance data from multiple sources. To be effective, the design of such a monitoring solution must allow for seamless integration into HPC applications and distributed services alike, be scalable, operate with a low overhead, and take advantage of the HPC platform. We propose SYMBIOMON, a monitoring service that is built by composing high-performance microservices. We describe its design and implementation within the context of the Mochi framework. SYMBIOMON combines a time-series data model with existing Mochi data services to collect, aggregate, and export performance metrics in a distributed manner. SYMBIOMON enables seamless, low-overhead monitoring and analysis of data services and HPC applications alike. Using HEPnOS, a production-quality Mochi data service, we demonstrate the use of SYMBIOMON to identify better service configurations.