Best Practices for Scalable Power Measurement and Control
Scott Walker, Marty McFadden · 2016
There are thousands of specialized registers on modern processors which provide useful features such as power budgeting, thermal monitoring, and performance counting. These registers fall into two categories, model specific registers (MSRs) and configuration space registers (CSRs). Many of these registers, such as those supported by PAPI, help the high-performance computing (HPC) community analyze their program in order to maximize performance and use resources more efficiently. However, there are many MSRs and CSRs which are not supported by existing performance tools. The MSR kernel module provides access to all MSRs, but requires the user to be root. Users will typically want to access a handful of MSRs sequentially, but we found that the existing MSR module has far too much overhead. Just like MSRs, users need elevated privileges to access CSRs with utilities such as lspci. Simply allowing users to access all of the registers is out of the question because of the security risks involved. Furthermore, using these registers requires detailed knowledge of the architectural changes in addition to the manufacturer's proprietary ways of encoding the data. In this paper, we will describe a group of utilities developed at Lawrence Livermore National Laboratory to address these problems. Our Libmsr API solves the usability issues by providing a simplified interface to common tasks. The companion kernel module, MSR-SAFE, allows white listing MSRs for userspace access, plus provides an optimized way to access MSRs in batches. Our CSR-SAFE kernel module is a first of its kind utility to white list sections of the PCI configuration space, where CSRs reside, for userspace access. We demonstrate the abilities of Libmsr and MSR-SAFE by using the utilities to set various power limits on the NAS parallel benchmark MG. Experiments analyzing our optimizations show a maximum speedup of 26X, and approximately 2-8X on average for typical batches versus the stock MSR kernel module. Our utilities currently support most modern Intel processors.