A Systematic Approach for Merging Small Files in Hadoop using Prolonged HDFS Framework
R Rathidevi, R. Parameswari · SSRN Electronic Journal · 2018
Hadoop is an open source frame work which works on java platform. It contains two major components HDFS (Hadoop Distributed File System) and Map reduce for processing large amount of data. The large amount of small files can be generated by various platforms. It’s difficult to handle small files because it takes lots of seek and hops while searching file from the block in the distributed system. The objective of this paper is developing an efficient approach improved HDFS to handle files from various users and to develop an effective method based on Merging files concept. By merging various small files into a large file reduces the memory utilization in name-node of HDFS. The proposed technique used to store and fetch the small files in a better manner. Intermediate Memory cache is used to Merge the files.