Designing an InfiniBand Metric Collector and Exploring InfiniBand Management Overhead and Scalability
Sjur Tveito Fredriksen · NORA - Norwegian Open Research Archives · 2017
The InfiniBand (IB) communication standard has over the last decade emerged to be one of the most popular interconnects used in high- performance computing. Deployment of IB networks have grown in terms of size and scale, and it is becoming a challenging task to keep track of the behavior of these networks. Monitoring of the performance of an IB network and overseeing errors and problems that might occur in the network is critical for the network administrators. There are already some tools that can help the network administrators, but many of these tools are old, and the user interface has failed to evolve with the network administrators expectations. Furthermore, little research has been done on how monitoring of an IB network influences application traffic or on how well switches handle repeated in-depth querying of their performance counters. The first goal of this thesis was to build a robust, efficient, and scalable IB performance monitor plugin for the Fabriscale Fabric Manager and monitoring software suite. The developed monitoring plugin swipes the network continuously and updates the Fabriscale Fabric Manager with metrics almost instantly so that network characteristics can be presented in real-time to the network administrators using a modern web-based graphical user interface. The developed plugin is of great value to Fabriscale by offloading the Fabriscale Subnet Manager, and it helps to reduce the time needed to reconfigure devices in a subnet when a fault is occurring. Thus the plugin helps network administrators to minimize downtime and to get improved utilization of the cluster. The second goal of this thesis was to inspect how fabric monitoring impacts the network, and how the network is affected by the metric collection. That is, this thesis looked on how multiple switches from multiple vendors handle in-depth querying of performance counters. It investigated how much extra bandwidth is generated by the monitoring software developed in this thesis, and how this can scale with the size of the network. Information about how the monitoring impacts the fabric is useful for Fabriscale and in turn network administrators when configuring the monitoring. We found that using in- band collection of metrics produces a minuscule amount of overhead and that the switches are handling being queried for their performance counters as often as every few milliseconds well.