Sort-Last In-Transit Data Visualization with SENSEI, Catalyst, and Unreal Engine
Isaac Nealey, Nicola Ferrier, Joseph A. Insley, Victor A. Mateevitsi, Silvio Rizzi, Jürgen P. Schulze · 2022
We present a end-to-end in-transit workflow that uses SENSEI [1], Paraview Catalyst [2], and Unreal Engine [3] for interactive rendering of large scale distributed simulation data as it is generated. It is designed for high performance computing (HPC) environments where many machines collectively model a system's behavior and each rank updates a partition of the problem domain. We consider the case where full-resolution post-hoc analysis is prohibitive or impossible, and the 3D metadata extracted in-transit (such as iso-surfaces or streamlines) is still prohibitively large and/or complex for rendering at interactive frame rates. Data is passed in-memory between each stage of the pipeline, with expensive disk I/O only occurring at the final stage, should a user wish to save their results. As video games have increased in complexity, developers have turned to procedural generation of 3D models and textures in order to algorithmically generate game content, like foliage or landscapes, without the need for manual modeling. We leverage these new tools in order to import the time-varying data simulated by domain scientists, and to manipulate these assets at runtime in order to update geometry as the simulation runs and more metadata is extracted.