Exploring the Reference Management in Parallel De-Duplication
Rui Zhu, Lei Hua Qin, Jing Li Zhou · Applied Mechanics and Materials · 2013
As the explosion of digital information in network, data de-duplication has been widely used in most backup systems to improve space efficiency. When only unique data segments are stored and shared by backup files, the reference information between the files and their data segments is becoming more and more important to track the data usage and reclaim of freed space. However, as the usage of multi-core and many-core, genereal Group Mark-and-Sweep method has some poor performance in concurrent reference updates due to the synchronization overhead. To alleviate this challenge, a Parallel Mark-and-Sweep mechanism has been exploied based on the research of DHT and similrity method. In our experiments with real-world datasets, it has shown a better performance comparing to the sequential method.