Erasure code of small file in a distributed file system
Xinhai Chen, Jie Liu, Peizhen Xie · 2017
With the development of Internet applications, the data storage of many applications has some new characters. There always exists a huge amount of pictures in distributed file systems, and the size of these pictures is usually no more than 1M bytes. To meet the demands in small file storage system which contains so many pictures, we provide a kind of distributed file system to store large amounts of small files. In our system, small files (actual data files) are merged into large files (default 64M) which we call block, and we use two-indexed way to reduce the pressure of nameserver, which maintains the information of correspondence between blocks and dataservers. The result shows that the memory usage of block information in nameserver is less than 2G when the system capacity is 1PB. It shows good scalability. Meanwhile, to reduce the cost of storage, we introduce the technique of erasure code which is an alternative offers the same data protection but reduces significantly the storage consumption. After grouping, the cost of storage dropped by 25% compared with 2 replications.