Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs
Anwar Hossain Zahid, Ignacio Laguna, Wei Le · 2024
As scientific codes are ported between GPU platforms, continuous testing is required to ensure numerical robustness and identify potential numerical differences between platforms. Compiler-induced numerical differences can occur when a program is compiled and run on different GPUs and compilers, and the numerical outcomes are different for the same input. We present a study of compiler-induced numerical differences between NVIDIA and AMD GPUs, two widely used GPUs in HPC clusters. Our approach uses a random program generator (Varity) to generate thousands of short numerical tests in CUDA and HIP, and their inputs; then, we use differential testing to check if the program produced a numerical inconsistency when run on NVIDIA and AMD GPUs, using the same compiler optimization level. We also use the AMD’s HIPIFY tool to convert CUDA tests into HIP tests and test if there are numerical inconsistencies induced by HIPIFY. In our study, we generated more than 600,000 tests and found subtle numerical differences occurring between the two classes of GPUs. We found that some of the differences come from (1) math library calls, (2) differences in floating-point precision (FP64 versus FP32), and (3) converting code to HIP with HIPIFY.