Implementation and Evaluation of a Cache Deduplication Mechanism with Content-Dened Chunking
Jun Murakami, Shun Ishiguro, Yoshihiro Oyama · 2012
Some application programs in data-intensive science create many large files that contain redun- dant data. Gfarm is a global distributed file system that enables an efficient use of large-scale data. In this paper, we describe the design and implementation of a cache deduplication mechanism with Content-Defined Chunking (CDC) for Gfarm, and report experimental results for evaluating the mechanism. CDC is a method of dividing a file into variable-size blocks (chunks) based on the contents of the file. The client stores the chunks in the local file system as cache files, and reuses them in the following file accesses. Deduplication of chunks reduces the amount of transmitted data and storage consumption. We confirmed through experi- ments that the proposed mechanism improved the performance of file read and the overhead of file write is about 30 % of the original performance.