A new metric for ranking high-performance computing systems
Jack J. Dongarra, Michael A. Heroux, Piotr Łuszczek · National Science Review · 2016
Performance benchmarks play an important role in various stages of hardware development and use. During development, hardware designers use benchmarks as proxies for full-fledged applications because prototype systems have a limited set of software tools that the applications require for compilation and execution. At procurement time, benchmarks serve as an assurance test that establishes the new system's viability and fulfillment of contractual obligations between the system integrator and the customer. And finally, benchmarking during the system's daily use can ensure proper operation of a computer installation, exposing any potential problems to the system's administrators, and gives the users an estimate of what their applications of interest can achieve without requiring the effort of building those applications, their software dependences, and loading the necessary input data that may initially reside offsite. The LINPACK benchmark [1] has been in continuous use since the 1980s. It was born out of necessity in the 1970s, when it was used to quickly test the performance of vector subroutines—which could serve as a good approximation of performance for the rest of the LINPACK library. Because of the nature of the implementation, the benchmark could also be used as a first-order approximation of other codes, partially due to the well-balanced hardware that offered plentiful bandwidth for every floating-point operation. Over the years, Moore's law eroded the compute-to-bandwidth balance, resulting in a memory wall. Today, this wall can be exhibited by the Intel Haswell (Xeon E7 v3 8900) processors, which feature 18 cores—each equipped with dual floating-point units (FPUs) with AVX vectors capable of fused multiply-add (FMA) instructions and clocked at 2.5 GHz (before TurboBoost)—bringing to bear, altogether, over 700 Gflop/s worth of peak performance—over 90% of which may be realized in a well-tuned matrix–matrix multiplication from a vendor library (MKL). At the same time, the memory controller for that processor has a theoretical maximum of 25 GB/s. The exact number of floating operations per every byte of bandwidth is 28.8—a far different ratio than the 1 flop per byte, which was the design point in the 1990s when the TOP500 list started ranking the supercomputers using the scalable version of the LINPACK benchmark. To reassess the application needs in this new and drastically different hardware regime, it is worthwhile to look at the computational simulations that drive national interest. Many of these simulations involve heat diffusion, electromagnetics, and fluid dynamics. Unlike LINPACK, which tests raw floating-point performance and its delivery through the BLAS API, these real-world applications rely on partial differential equations (PDEs) that govern the continuous representations of the physical quantities such as particle speed, momentum, etc. These PDEs involve sparse (not dense) matrices that represent the 3Dl embedding of the discretization mesh. While the size of the sparse data fills the available memory to accommodate the simulation models of interest, most of the optimization techniques that help achieve close to peak performance in dense matrix calculations are only marginally useful in the context of sparse matrices originating from PDEs. Our new benchmark, called high-performance conjugate gradients (HPCG) (further information is available at www.hpcg-benchmark.org), is based on Mantevo collection's HPCCG code base [2], but aims to go beyond its originator and represent the calculations that commonly occur during the numerical solution of PDEs in modern state-of-the-art solvers. To that end, HPCG is dominated by sparse operations such as sparse matrix-vector product and sparse matrix triangular solve. When properly implemented, these operations stress the memory subsystem, including the higher level cache memories’ ability to coalesce irregularly strided memory loads, bandwidth and latency of the main memory, as well as the processing units’ ability to schedule around the inherent delays of the main memory. In terms of communication, the benchmark tests global and neighborhood collectives for dot products and halo (domain boundary) exchanges, respectively. As an added irregularity, HPCG features elements of multigrid with a local smoother. This counteracts the initial regularity of the domain and increases the complexity of memory access patterns that become much less bandwidth bound and more latency bound at the coarse levels of the multigrid. As the figure of merit, HPCG uses the Gflop/s rate based on the apparent (rather than actual) number of floating-point operations executed during execution (apparent operations are those that have to be performed based on the structure of the matrix, actual operations depend on the implementation—they could be larger if the implementation chooses to recomputed some of the intermediate results). Such a metric is unconstrained and benefits from improvements applied to any of the hardware components relevant to the simulation applications based on PDEs. At the same time, the low value of the achieved Gflop/s serves as a stark reminder of the imbalance between the floating-point capability and the speed of various data storage and transmission pathways in modern supercomputing systems. The HPCG Benchmark can help alleviate many of the problems described above using the following principles: Provide comprehensive coverage of the code types that test major communication and computational patterns. Both global and neighborhood collectives are essential in terms of communication, and the important computational patterns include vector updates, dot products, sparse matrix-vector multiplication, and local triangular solves. These are inspired by our production differential equation codes, both implicit and explicit, and therefore made their way into this benchmark. In addition, emerging asynchronous collectives and other latency-hiding techniques can be explored in the context of HPCG and aid in their adoption and optimization on future systems. Represent a minimal collection of the major patterns. Even though it is a benchmark code rather than a full application, HPCG represents major computational patterns well, and is—to our knowledge—the smallest code containing them. The HPCG methods and algorithms approximate real mathematical computation very well, which aids in our validation and verification efforts described below. Reward investment in high performance of collective communication primitives. The neighborhood and all-reduce collectives represent essential performance bottlenecks for our applications. Implementations of these primitives can benefit from high-quality system design. As a consequence, improving the performance of HPCG will improve the performance of real applications. Reward investment in local memory system performance. As the local shared-memory and multicore performance of HPCG is largely determined by the effective use of the local memory system, it becomes important to stimulate improvements at the hardware level by rewarding good design decisions with appropriate improvements in the benchmark results. A good correlation already exists between improvements in the implementation of HPCG data structures, compilation of HPCG code, the performance of the underlying system, improvements in HPCG benchmark results, and real application performance. These results will inform application developers of new approaches to optimizing their own implementations. For a long time, the standard for measuring parallel sparse solvers was the NAS parallel benchmarks (NPB) collection [3,4] and its recent updates [5]. Since HPCG and NPB both aim to extract the most common features of scientific computational kernels, there are similarities in design and algorithmic implementation between both. There are, however, three important differences: (i) the data distribution and generation, (ii) splitting of code components, and (iii) algorithmic goals and choices of the code. While NPB uses 2D data distribution, HPCG uses 1D distribution—a more common distribution for modern solvers. NPB uses the structure of the matrix chosen from a uniform random distribution based on a multiplicative random number generator [6]. This can hardly be correlated with the actual matrices of common PDE discretizations that admit a 3D embedding or similar low-dimensional space representation. Thus, HPCG uses a regular 3D grid discretization (see below), but prohibits exploiting this structure (this may be enforced by stochastically testing the implementation with slightly perturbed structure). NPB splits the multigrid (NPB MG) and conjugate gradient (NPB CG) components, while HPCG treats them together. Additionally, HPCG combines domain decomposition with additive Schwarz coupling as that is the preferred method for a parallel sparse solver. Another sparse benchmark, which in our mind is very closely related to our efforts, was the iterative solver benchmark [7]. The algorithmic scope of this benchmark is broader than HPCG: it includes both CG and GMRES methods, it tests a number of meaningful sparsity patterns, and uses several preconditioners. With such a broad spectrum of tested codes it still lacks any tests of scalability that target distributed memory parallelism, multicore processors, or hardware accelerators—testing of each of these aspects of modern supercomputers is an essential prerequisite for a comprehensive benchmark code. The model problem that HPCG benchmark solves is a discretized Poisson PDE in three dimensions [8]. The iterative method of choice is preconditioned CG applied to the resulting symmetric positive definite sparse linear system. The original PDE models a single degree of freedom heat diffusion equation with zero Dirichlet boundary conditions. The PDE is discretized with a finite difference scheme on a 3D rectangular grid domain with regular spacing of the nodes, thus producing a sparse matrix with a simple and predictable structure. The parallelism of the solver may be scaled arbitrarily through the domain decomposition scheme with additive Schwarz coupling. The partitioning of the global discretization grid among the distributed memory processes is regular and three dimensional: across the x-, y-, and z-axes. The factoring of the distributed processes (the ranks) into a 3D regular grid can deteriorate into a 1D or 2D structure if the total rank count does not have good integer factors, e.g. it's a prime number. In such a case, the communication patterns between the processes will be local for the most part, and will not reflect the challenges imposed by scientific applications on the large-scale HPC networks. For this reason, one of the checks in HPCG keeps the ratio min(x,y,z)/max(x,y,z) high enough: it is not supposed to drop below 0.125. Choosing this ratio to be high would keep the shape closer to a cube, but due to the distribution of factors in process counts up to hundreds of millions, keeping the ratio high would preclude a large number of configurations from being eligible. During the setup phase, a logically global, but physically distributed, sparse linear system is constructed using a 27-point stencil at each grid point in the 3D domain, such that the equation—at any interior point—depends on the values at that point and its 26 surrounding neighbors. The matrix is constructed to be weakly diagonally dominant for the interior points of the global domain, and strongly diagonally dominant for the boundary points, reflecting a synthetic conservation principle for the interior points and the impact of zero Dirichlet boundary values on the boundary equations. The resulting sparse linear system has the following properties: A sparse matrix with 27 non-zero entries per row for interior equations and 7–18 non-zero terms for boundary equations. A symmetric, positive definite linear operator. The boundary condition is reflected by subtracting 1 from the diagonal. A generated known exact solution vector with all values equal to 1. A matching right-hand-side vector. An initial guess of all zeros. The preconditioned CG method, shown in Fig. 1, allows the code to maintain the orthogonality relationship with a short three-term recurrence formula. This in turn allows the linear system data to be scaled arbitrarily without worrying about the excessive growth of storage requirements for the orthogonal basis, unlike GMRES, which was used in the iterative solver benchmark. GMRES internal storage requirements grow with the system size and it requires to balance the scale with the restriction inherent to the solution method. CG uses short-term recurrence relation to keep track of its progress and hence internal storage does not grow with the problem size. As mentioned above, the central purpose of defining this sparse linear system is to provide a rich vehicle for executing the collection of important computational kernels. The pseudo code shown in the figure features these kernels prominently. At the same time, the benchmark's primary function is not to compute a numerically exact solution to the model problem. In fact, the iteration counts are fixed in the benchmark code and we do not expect convergence to the solution, regardless of the problem size. We do use the spectral properties of both the problem and the preconditioned CG algorithm as part of software CG algorithm used by HPCG of using the same initial guess each These chosen to be large to test the system's and ability to we can the numerical results for at the of each of the iteration The method has been a of and may be for PDEs. by the it is to it to a much larger of linear and PDEs As described HPCG all of the computational and communication patterns exhibited by solvers. the include through methods, such as the and the dominant performance at coarse grid levels in the of latency rather than the are the at the grid levels and dot products of the preconditioned CG method. For these with version of an was in the code to model the of In the method is used for as shown in Fig. 1 with the levels of and as well as restriction and The problem when this of new was a potential for a in the code complexity and growth in and components, such as the validation and verification to the impact of the we the components and them in terms of commonly used of a solver. The for all of the levels of our is a local The and with the restriction and is by simple of the number of points in every and thus each coarse grid level has as points as the grid as was the for the preconditioned CG from Fig. 1, our is only to provide components rather than a a comprehensive implementation of a full solver. we include the full which are the shape of the grid In we do not an at the grid level to all and provide a good solution vector for we the number of grid levels to which results in a in the number of grid points, which is to most of the bandwidth latency bottlenecks and the performance of common algorithmic in used solvers. At bandwidth is the main as the number of grid points is large for to the When the coarse grid is the number of points is and latency is this many to more grid levels could these bottlenecks more but as the grid becomes the latency becomes one of many other bottlenecks such as due to of benefits from In this limited implementation, we also the patterns of code execution and the integer to some of the inherent in grid As was this the aim of any HPCG in to a balance between software dependences, and requirements imposed by real scientific applications. The implementation and software play a role in and we made design choices these For the we a of modern which is a and the to the progress of the standard and include features that are not to it to the in the if at As of this the of have a of the and to an using some of these more modern features could the of the code. At the same time, the code base is simple by design and the implementation and without on while the use of in scientific As a point of interest, a similar to any other of choice or the standard library of these we for and for distributed memory The is in version while the is in version Both of these are up by of continuous development and It is to that these will and be available for future hardware are an important of modern supercomputing and To this end, the code available for has for that of such and In addition, there are available from the HPCG or from the The regularity of the model discretization grid to the sparse data structure for to and the points to achieve good balance, communication and good local performance. we that such would the of the benchmark and its results. we that the of the regularity not be into when and optimizing the code for the To that the discretization be as a without any properties known a In the users may of the of the to problems with their since many aspects of the solution are known in and can serve as a useful we the use of of the problem when the CG We however, that users may to use the of the spectrum of the discretization matrix to estimate the of their solver. HPCG includes a code that and from because it is that future computer systems will not be to provide execution for floating-point In fact, may already be on some and can be by code on some This may in because floating-point is not thus we may not have when the same exact computation on the same number of of the same system. This is in with many of our applications it a to applications that their computational results and numerical in the of The setup and execution of the tests in HPCG the from to To during the iteration HPCG code uses standard software such as and These are to a of that in when an version of the benchmark. In to full of the tested the is not the computational kernels in The code that we provide is on and which may have on performance for a system. In we have already to the HPCG code by benchmarking and vendor with and good results The resulting code is available on the or from the vendor In to the kernels, HPCG includes a test for the sparse matrix with a discretization matrix and for the symmetric smoother. a spectral test is also in purpose is to test for convergence of the CG algorithm on a matrix that is close to being diagonal. the theoretical and to the convergence underlying the CG solver we that a fixed number of is for such and the by the this The spectral test is to potential in the implementation related to calculations and to convergence rate due to matrix The main performance by HPCG is the number of floating-point operations per It is a metric that is bound at by the peak performance rate HPCG a single of the which is related to the number of their and the available such as and This bound is for iterative solvers for such as or most other scientific applications. the LINPACK benchmark can data and in its computational patterns to be to keep the at every and thus extract a of the theoretical peak performance of the For an of such an implementation of to the TOP500 of the performance is largely on the memory and the of the The and parallelism in data all to an all good by It is to look at these factors the supercomputing in the In fact, this is the purpose of and the HPCG results on large supercomputing around the the HPCG results, from supercomputing from all over the as in and and in Since HPCG is still a new benchmark, not all of results are of the same and the number of entries does not the to this growth rate and the benchmark adoption in it is worth the between the of LINPACK results and the of the TOP500 There are large systems results are in the HPCG and a number of may be on the results The HPCG results of difference across the systems on a single similar to what has been on the TOP500 Another worth is that a number of systems on the HPCG list are than their from This is due to the results being scaled across larger of the and some of the results being due to code and In addition, the recent at the of the TOP500 which in the several can also be on the HPCG results HPCG results since Another metric to help systems is to count the number of between the TOP500 and HPCG is the that between the and the While the is on the TOP500 the is on the HPCG It is worth that both systems use an version of the and HPCG codes, which performance results. peak performance is to computer due to the use of from and closely the peak performance only of the peak while computer over The for this difference in the of the uses while uses The is with to the local performance while the while still to many much When the of increases as the of execution communication and neighborhood is As a computer and The to over of the peak performance while the is at about of it is important not to much into the total count for the HPCG the HPCG benchmark more establishes its in the and more can be made across more comprehensive set of results. at the of the peak performance that is achieved by various of systems on the HPCG we three main results. with very high memory bandwidth achieve about of the peak performance. with very good achieve about while more systems have the ratio at around This is similar to the on the TOP500 list systems with achieve about of the peak performance and only a between performance for the and The figure is to the HPCG which allows to with to the peak and Another is a drop of in performance from to between and HPCG performance for the TOP500 from it is also worth the main performance for the supercomputing system. peak performance rate is and the benchmark about of that number at At the other of the with PDE HPCG only still the in the by this