Multi-modal campus one-stop data storage optimization method for massive small files
Hong Zhang, Yang‐Yang Chen · 2023
The Hadoop distributed file system (HDFS) is usually used for storing and managing large files. When storing and computing large amounts of small files, NameNode memory and access time are consumed, which is an important factor restricting HDFS performance. Aiming at the problem of massive small files in multi-mode campus one-stop data, a storage optimization method of massive small files based on double-layer hash coding and HBase is proposed. When small files are merged, the extensible hash function is used to build index file buckets so that index files can be dynamically expanded as required to achieve file merging and index functions. The experimental results show that compared with the original HDFS, HAR, MapFile, TypeStorage and HPF small file merging methods, the proposed algorithm has a shorter file reading time, and can improve the overall performance of HDFS when processing massive small files in multi-modal campus one-stop data.