Scaling Study of Flow Simulations on Composable Cyberinfrastructure
Sambit Mishra, Freddie Witherden, Dhruva K. Chakravorty, Lisa M. Pérez, Francis Dang · Practice and Experience in Advanced Research Computing · 2023
Cyberinfrastructure (CI) systems employing composable approaches give researchers the capability to define resources best suited to meet the needs of their computational workflows. Among these approaches, composing disaggregated computing resources over a software-defined network offers the promise of supporting workloads requiring dynamic access to a large pool of accelerators or memory. Much remains to be understood about the architecture-induced constraints of this approach, and how it impacts scientific and engineering applications software. Here, we study the performance of the highly scalable open-source flow solver, PyFR, in a GPU-based composable environment orchestrated using a software-defined PCIe Gen4 fabric. PyFR emphasizes communication between GPUs and helps understand how GPUs on disaggregated resources can be optimally configured for performance. Strong-scaling and weak-scaling performance studies on composed configurations are compared to a traditional CPU-GPU cluster with InfiniBand interconnect. Factors affecting performance, and the need for new benchmark suites for composable devices are discussed.