New Lustre features to improve Lustre metadata and small‐file performance
John Fragalla, Bill Loewe, Torben Kling Petersen · Concurrency and Computation Practice and Experience · 2019
Summary As HPC I/O evolves beyond the challenges of large I/O performance, several new Lustre features ‐ Distributed Namespace (DNE) 2, Progressive File Layout (PFL), and Data on Metadata (DoM), when combined with flash‐based Lustre targets, can provide metadata and small‐file performance improvements to applications transparently. To improve the execution of single‐directory operations and increase the scalability of overall metadata operations, Striped or Remote Directories with DNE2 can be configured with multiple MDTs. For small‐file I/O, Progressive File Layout with flash and disks can be used to optimize small‐ and large‐I/O by seamlessly storing files on flash‐based MDTs or specific flash OSTs. This paper will share performance results that demonstrate the scalability benefits of metadata performance using striped or remote directories as well as small‐file performance with DoM and Flash OSTs with PFL, both with and without Cray's block I/O accelerator (NXD) configured.