Conduit: A Successful Strategy for Describing and Sharing Data In Situ
Cyrus Harrison, Matthew C. Larsen, Brian Ryujin, Adam Kunen, Arlie G. Capps, Justin Privitera · 2022
Data representation and coupling between scientific libraries is a key challenge to building a vibrant ecosystem of HPC simulation tools. From bespoke data structures to hundreds of file-based data models, the myriad of possible choices involved both enables key features and blocks adoption of others. Connecting data between code bases requires agreeing on or adapting between data representations. While in some cases this process is trivial, for more complicated cases, adapting data becomes a costly barrier. Conduit was designed within this context to help meet the key challenge of sharing data across HPC simulation tools by providing a dynamic API to describe in-memory data. It supports coupling simulations and connecting simulations to analysis and I/O libraries.Conduit is an open source project from Lawrence Livermore National Laboratory. It started in 2013 and has evolved through co-design with simulation applications and in situ tools since. Conduit is now an established part of LLNL’s simulation data management strategy and has been adopted as the mesh-data interface for DOE community in situ visualization tools. While Conduit has been discussed briefly in prior in situ research, this paper provides a broader overview of Conduit, background on the evolution of the project, and details on recently added features relevant to in situ use cases.