Efficient Metadata Indexing for HPC Storage Systems
Arnab Kumar Paul, Brian Wang, Nathan Rutman, Cory Spitz, Ali Raza Butt · 2020
The increase in data generation rate along with the scale of today's high performance computing (HPC) storage systems make finding and managing files extremely difficult. Efficient file system metadata indexing and querying tools are needed to ease file system management. Current metadata indexing techniques either use spatial trees or an external database to index metadata. Both approaches have their drawbacks which reduce the performance of indexing and querying the metadata on large scale file systems. In this paper, we have developed Brindexer, a metadata indexing and search tool specifically designed for large-scale HPC storage systems. Brindexer is mainly designed for system administrators to help them manage the file system effectively. It uses a leveled partitioning approach to partition the file system namespace, and has an in-tree design to reduce resource utilization from an external database. Brindexer uses RDBMS for efficient querying of the metadata index database, also uses a changelog-based approach to effectively handle real-time metadata changes and re-index the metadata at regular intervals. We implement and evaluate Brindexer on a 4.8 TB Lustre store and show that it improves the indexing and querying performance by 69% and 91% when compared to state-of-the-art metadata indexing tools.