An Algorithm of Merging Small Files in HDFS
Xianzhen Ren, Xiuhua Geng, Yi Ming Zhu · 2019
Small files are files that are much smaller than configured HDFS block size (by default 64MB or 128 MB on most up-to-date Hadoop installation). HDFS is meant for storing large files, but due to input source and ingestion methods large number of small files need to be stored in HDFS, and HDFS cannot handle small files in an effective and efficient way as it is designed to read large chunks of data in a sequential way. This paper describes one algorithm of merging small files into one container file than can resolve the issue.