Deliverable D5.1 Report on the ESDM runtime extensions for parallel in-flight analytics
Donatello Elia, Cosimo Palazzo, Manuel Musio, Reinhard Budich, Oliver Heidmann, Dela Spickermann, Julian Martin Kunkel · Zenodo (CERN European Organization for Nuclear Research) · 2021
As the volumes of data produced by Earth-System Models (ESM) continue to increase, solutions capable of addressing post-processing, analysis and visualisation (PAV) of these data at extreme-scale are required. Novel technologies supporting parallel data I/O and processing for different scientific domains are, thus, being developed. For example, the Earth System Data Middleware (ESDM), developed in the context of the ESiWACE CoE for climate and weather applications, aims to provide improved I/O towards heterogeneous storage back-ends. The ESDM-PAV runtime, reported in this document, provides the extensions to the ESDM middleware with the goal of specifically supporting PAV needs in the field of climate and weather applications. It delivers a simple high-level interface to build PAV experiments composed of different tools, as well as integrating with the HPC software ecosystem. It is built on top of the ESDM software used for I/O and in-flight analytics purposes. The proposed solution allows the support of different types of execution parallelism, exploiting a task-based programming abstraction, whereas the in-flight analytics extensions allow performing the analysis while the data are being accessed, potentially offloading the processing on the storage systems, therefore reducing data movement between storage and computing resources. This deliverable provides the formal documentation of the final release of the ESDM-PAV runtime: its complete design, the parallelisation strategies employed and the software implemented. It describes the approach considered to support efficient in-flight analytics on top of the ESDM, as well as the integration strategy for PAV applications. It also presents a set of usage examples of the supported features and the related system documentation.