Datanode Optimization in Distributed Storage Systems
Xiaokang Fan, Shanshan Li, Xiangke Liao, Lei Wang, Chenlin Huang, Jun Ma · 2012
Abstract—Distributed storage systems designed for small files have been developing rapidly, like Facebook’s hayStack, Twitter’s Cassandra and so on. But, under our observation, there are still some drawbacks in these systems. For example, they do not have cache specified for files and have not taken the relationship inherent in application-specific knowledge between files into consideration. We propose a file-level cache on datanode and co-location of affinitive files based on application-specific knowledge. We use a synthetic data set and a real world trace to evaluate our optimization. The file-level cache and co-location of affinitive files together can improve system’s throughput by 20%-50%. Keywords-distributed storage system; file-level cache; co-location. I.