An Optimized Storage Method for Small Files in Ceph System
Chao Li, Tengfei Wang, Lei Cao, Taotao Xie · 2023
In the scene of massive small files, there are a lot of data holes and redundant metadata in the distributed storage system based on Ceph, resulting in a waste of storage space. And the network consumption, PG(placement group) mutex and IO order preservation in the IO path will affect the IO performance. In order to solve the above problems, an online small file merging storage method is proposed. This method uses the relevance judgment module based on Crush (controlled replication under scalable hashing) to group the small files, and merges the small file data in the request context. When merging, the CV(condition variable) pool method is used to synchronize the threads in the same merging group. After merging, small files are stored sequentially and share the same metadata. In addition, this method optimizes the deletion of merged small files, and only records GC(garbage collection) logs when deleting, and garbage data is cleaned asynchronously by GC threads. The experimental results show that this method can improve the writing performance of small files by 62%, reduce data holes by 80% and reduce metadata by 50%.