Performance Analysis of Small Files in HDFS using Clustering Small Files based on Centroid Algorithm
R Rathidevi, R. Parameswari · 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · 2020
In day to day life, a lot of files are generated from various areas, due to the rapid development of technologies. Storing these files consumes a lot of memory space. Large-sized files are not only represented as Big Data. Large numbers of small files are also considered as big data. To process large-sized file Hadoop is used. Processing small files in Hadoop is not easy, because it holds memory space of size 128MB separately for each and every dataset. To overcome this, Clustering Small Files based on Centroid (CSFC) approach is used to place the related files in a cluster. If the fetched data is not related to any other files they knew its different and a cluster will be generated. The combined files are forwarded to HDFS for further processing. The Name node holds metadata and Data Node hold the dataset. The data set can be fetched directly from the Data node in HDFS efficiently.