ESSA 2022 Invited Speaker DAOS: Nextgen Storage Stack for HPC and AI
Johann Lombardi · 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · 2022
DAOS is an open-source scale-out object store designed from the ground up to deliver extremely high bandwidth/IOPS and low latency I/Os to the most demanding data-intensive workloads. It aims at supporting nextgen scientific workflows combining simulation, big data and AI in a single storage tier. DAOS presents a rich and scalable storage interface that allows efficient storage of both structured and unstructured data. DAOS supports multiple application interfaces including a parallel filesystem, Hadoop/Spark connector, TensorFlow-IO, native Python dictionary bindings, HDF5, MPI-IO as well as domain-specific data models like SEGY. Many DAOS deployments are underway including a 230PB installation connected to the ALCF's Aurora system and a 1PB DAOS system for LRZ's SuperMUC-NG phase 2.