Zettafly: A Network Topology with Flexible Non-blocking Regions for Large-scale AI and HPC Systems
Dezun Dong, Ziyu Wang, Fei Lei · 2025
Interconnection networks are playing an increasingly crucial role in achieving scalability and throughput for post-exascale and zettascale computing systems.Resource consumption characteristics of AI and HPC workloads are essential to designing effective network infrastructure.The operation practice of production supercomputing systems reveals that small and medium-sized jobs consume most compute-core hours, generating traffic that mainly utilizes a size-restricted portion of the network.Unfortunately, current topologies lack sufficient support for flexible partitioning of distinct concurrent jobs.This study bridges this gap by exploring a new tradeoff among scalability, throughput, and non-blocking regions.We present Zettafly, a family of low-diameter topologies with large-scale non-blocking sub-networks, allowing the majority of jobs to be isolated within a single sub-network.Zettafly is a two-layer structure, consisting of non-blocking groups and global routers, such that each minimally routed packet between groups traverses at most one global router.We also introduce simple and efficient adaptive routing algorithms for Zettafly.We conduct extensive simulations and analysis to evaluate the performance and cost of Zettafly against state-of-the-art topologies.The results show that Zettafly achieves great performance under typical HPC workloads, offers cost-effectiveness under multi-task mixed traffic, supports flexible configurations and incremental deployments, and is costeffective to deploy for post-exascale and zettascale systems.