A survey on deduplication systems
Amdewar Godavari, Chapram Sudhakar · International Journal of Grid and Utility Computing · 2024
With the arrival of new technological trends such as Big Data and Internet of Things, tremendous amount of duplicate data is being generated. Duplicate data causes the wastage of storage capacity and degradation of performance of the storage systems. Data deduplication is a storage optimisation technique that is used to eliminate duplicate data. Deploying deduplication system for primary storage or secondary storage is challenging due to extra latency incurred in deduplication processing. Apart from this, as duplicates are eliminated, deduplication affects contiguous placement of data on the disk, which is known as disk fragmentation problem. This paper gives overview of issues and solutions proposed for deploying deduplication component for primary and/or secondary storage systems with centralised or distributed approaches. Experiments are conducted using Destor tool on different data sets. The results are used to study the effect of different chunking algorithms on deduplication phases.