A Faster Read and Less Storage Algorithm for Small Files on Hadoop
Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuaifang Wang, Zhaoqun Sun · 2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI) · 2021
Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by thepair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.