Proactive Endpoint Congestion Avoidance in UCC
Ferrol Aderholdt, Aamir Shafi, Manjunath Gorentla Venkata · 2025
As distributed computing workloads scale to hundreds of thousands of nodes with millions of processes for both High Performance Computing (HPC) and Deep Learning (DL) workloads, the detection and mitigation of endpoint network congestion becomes an increasingly important problem. This can be accomplished reactively (i.e., congestion control) or proactively (i.e., congestion avoidance) and can be performed either at the switch or the endpoint Host Channel Adapter (HCA). Often these mechanisms make use of Round-Trip-Time (RTT), signals (i.e., Endpoint Congestion Notification (ECN)), or packet scheduling techniques to mitigate or avoid congestion and interact with the network with respect to all communication types including point-to-point and collective operations. Unified Collective Communication (UCC) is an API and collective library that focuses on providing a unified API for collective communication for various programming models and over various devices and device memories. However, it contains limited algorithmic solutions to provide endpoint congestion avoidance. This work focuses on highlighting the need for collective avoidance algorithms to be present in a collective library such as UCC and demonstrates the benefits of such an approach with a proactive credit-based congestion avoidant extension to collective algorithms for collectives such as Alltoall and Alltoallv. We evaluated our extensions on multiple systems with micro-benchmarks and applications. Our evaluation has shown up to 8X performance improvement for congested workloads utilizing these congestion avoidant algorithms on RoCEv2 networks.