From Islands to Archipelago: Towards Collaborative and Adaptive Burst Buffer for HPC Systems
Mingtian Shao, Ruibo Wang, Wenzhe Zhang, Kai Lü, Yiqin Dai, Huijun Wu · 2025
Modern supercomputers increasingly use node-local storage as burst buffers (BB) to address I/O bottlenecks.However, current BBs do not naturally support workflows, a common workload in HPC consisting of many interconnected subtasks.Workflow I/O can be divided into three types: intratask I/O, inter-task I/O, and stage-in/out I/O.While BBs can accelerate intra-task I/O, they often overlook the other two.Inter-task I/O relies on migrating data through the Parallel File System (PFS), which can slow down overall performance.Although allocating more resources to create larger BBs could help, it increases costs.Additionally, temporary BBs lack permanent storage, requiring data migration between the PFS and BB for stage-in and stage-out I/O.This process often involves multiple data copies and reduces I/O efficiency.Even for intra-task I/O, unbalanced data distribution can cause bottlenecks on heavily loaded nodes.To improve workflow acceleration in BB systems, it is important to address the needs of all the above-mentioned three