WisIO: Automated I/O Bottleneck Detection with Multi-Perspective Views for HPC Workflows
Izzet Yildirim, Hariharan Devarajan, Anthony Kougkas, Xian‐He Sun, Kathryn Mohror · 2025
Why I/O Bottlenecks Matter in HPC• Modern HPC workloads (AI, simulations) involve massive data transfers that are crucial for enabling scientific discoveries• The large volume of these data transfers often lead to workloads spending significant amount of time performing I/O• Recent studies show that it is between 25-40% of total runtime • As a result, tuning the performance of data transfers via I/O analysis has become a routine task for application developers 6/27/2025 High-Level Execution Flow of WisIO 6/27/2025 WisIO 7 • Transform raw trace data into multi-perspective views • File, process, timeline, or user-defined High-Level Execution Flow of WisIO 6/27/2025 WisIO 8 • Transform raw trace data into multi-perspective views • File, process, timeline, or user-defined High-Level Execution Flow of WisIO 6/27/2025 WisIO 9 • Severity-based classification using I/O metrics • Quantifies how "bad" an I/O behavior is via a relative severity angle High-Level Execution Flow of WisIO 6/27/2025 WisIO 10 • Severity-based classification using I/O metrics • Quantifies how "bad" an I/O behavior is via a relative severity angle High-Level Execution Flow of WisIO 6/27/2025 WisIO 11 • Explains bottlenecks via rule-based reasoning • Identifies one or more causes (e.g., small reads, metadata overhead) High-Level Execution Flow of WisIO 6/27/2025 WisIO 12 • Explains bottlenecks via rule-based reasoning • Identifies one or more causes (e.g., small reads, metadata overhead) Implementation & API 6/27/2025 WisIO 13 Implemented in Python, for versions 3.8 and above • Parallel and distributed via Dask Works out-of-the-box with trace data from common I/O monitoring tools • Darshan, DFTracer, Recorder Two user-facing interfaces: • CLI: Installable via pip, highly configurable • Python API: Allows interactive analysis Multiple output types: