Optimization of De-duplication Technology Based on CDC Blocking Algorithm
Junyang Nie, Lijuan Wu, Jingzhang Liang · 2019
All kinds of information data show exponential growth, and there are a lot of duplicate data in the mass data, which seriously affects the efficiency of data processing and causes serious waste of storage space. How to delete duplicate data has become a research hotspot in the current storage field. The content-defined chunking(CDC) algorithm is one of the most widely used data de-duplication algorithm, but it often produces too small or too large data blocks, resulting in low data processing efficiency and waste of storage space. In this paper, CDC chunking algorithm is optimized, and data processing efficiency and data de-duplication rate are improved in three experimental data sets through experiments.