Metadata considered harmful ... to deduplication
Xing Lin, Fred Douglis, Jimmy Li, Xudong Li, Robert Ricci, Stephen Smaldone, Grant Wallace · 2015
Deduplication is widely used to improve space effi-ciency in storage systems. While much attention has been paid to making the process of deduplication fast and scalable, the effectiveness of deduplication can vary dramatically depending on the data stored. We show that many file formats suffer from a fundamental de-sign property that is incompatible with deduplication: they intersperse metadata with data in ways that re-sult in otherwise identical data being different. We examine three models for improving deduplication in the presence of embedded metadata: deduplication-friendly data formats, application-level post-processing, and format-aware deduplication. Working with real-world file formats and datasets, we find that by separating metadata from data, deduplication ratios are improved significantly—in some cases as dramatically as 5.6×. 1