Investigation of Server Level Direct-To-Chip Two Phase Cooling Solution for High Power GPUs
Akshith Narayanan, Qingyang Wang, Serdar Ozguc, Richard W. Bonner III · 2024
Abstract As artificial intelligence and technology advance, top end server CPU and GPUs are getting more and more powerful. Conventional air-cooling struggles to keep up with the surging processor power densities, leaving room for an efficient and scalable solution. This investigation focuses on the Nvidia H100 GPU architecture in a mimic 1U server with four-way GPUs. While direct to chip two phase cooling offers high heat transfer coefficients and heat capacity, there are technical challenges associated with pressure drop and flow maldistribution to understand the requirements and cost of its use. A single sled test bench is built to conduct systematic experiments with in-house designed Thermal Test Vehicles (TTVs) that replicate the form factor of Nvidia H100 GPUs. The facility is used to characterize pressure drops, die temperature, pumping power and thermal resistance at varying heat loads and flow rates, using R1233zd(E) as the working fluid. Flow restrictors were carefully chosen to balance the flow between the chips, making sure each of them receives sufficient liquid in all usage cases. The results show that the thermal resistance of the cold plate assembly varies between 0.02–0.03W/K. Higher power tests, up to 1.35kW, were conducted to understand how future-proofed the cold plate design is, in terms of rapidly scaling heat loads in the market. Results show that with sufficient flow, the system operates extremely well. The flow restrictors chosen scale quadratically with pressure drop and mass flow rate, and they efficiently suppress the maldistribution between chips on the server level and can be expanded to balancing flow on the rack level. The investigation concludes that pumped direct-to-chip two-phase cooling offers an elegant and scalable solution to the rapidly increasing power densities of CPUs and GPUs, while maintaining efficiency and minimizing the leak risks compared with using a water-based solution.