Addressing Hadoop's Small File Problem With an Appendable Archive File Format
Thomas Renner, Johannes Müller, Lauritz Thamsen, Odej Kao · 2017
Hadoop has been used widely for data analytic tasks in various domains. At the same time, data volume is expected to grow even further in the next years. Hadoop recently introduced the concept Archival Storage, an automated tiered storage technique for increasing storage capacity for long-term storage. However, Hadoop Distributed File System's scalability is limited by the total number of files that can be stored, and it is likely that the number of files increases fast when using it for archival purposes.