Small Files in HDFS and Their Impact on Hadoop Performance
Christoph Schmitz, Gali Peled, Oded Koren · 2021
In this paper, we discuss the issues that arise with fragmentation of data in big data platforms such as Hadoop when small files are processed. We examine the conditions under which small files cause problems, and review some approaches that have been made to mitigate these problems. We conduct experiments regarding the interaction between fragmentation and batch processing performance, and argue that with some care in application design and configuration, most of the fragmentation problems can be avoided without extensive changes to the Hadoop infrastructure.