MOSM: An approach for efficient storing massive small files on Hadoop
Kun Wang, Yang Yang, Xuesong Qiu, Zhipeng Gao · 2017
Benefiting from its high scalability and high reliability, Hadoop has become a popular big data processing platform at present. Hadoop Distributed File System (HDFS) which is one of the cores of Hadoop can efficiently store large files. However, massive small files stored in the HDFS cause the “small files problem” due to the bottleneck of NameNode memory and access performance. To solve the defect for storing massive small files, we propose a multilevel optimization storage method (MOSM), which optimizes the storage process of small files. We use an algorithm to merge small files into large files to reduce memory utilization of NameNode. After merging, we design an efficient hybrid index strategy and a prefetching cache mechanism to improve the speed of small files accessing. The experimental results indicate that the MOSM is able to reduce the load of NameNode effectively and improve the ability of Hadoop cluster to store numerous small files.