Research on Small Files Optimized Storage Strategy in Hadoop System
DU Zhonghu · Intelligent Computer and Applications · 2015
With the advent of BIG data,big data processing platform such as Hadoop has emerged. But its storage carrier-- Hadoop distributed file system has many significant flaws on the storage of mass small files,storing massive amounts of small files will not only increase the load of entire cluster,but also decrease operating efficiency. In order to solve the defect,the usual method is to merge small files to a big one,and then it will be stored instead. However,the conventional method does not take advantage of the volume size distribution,so it failed to further enhance the combined effect of small files. This paper presents a data block based on a balance of small files merging algorithm to optimize distribution of merged large files volume,which could effectively reducing the HDFS data block. Thereby the reducing of primary node memory consumption and running load will cause data processing can be run more efficiently.