Entrodeduper: Data Deduplication Using Entropy and Secure Hash Algorithm
Tin Thein Thwel, Hsu Mon Maung · 2023
Being a data science era, any bulk of data concerned with specific fields need to maintain in data centers with the help of rack server. For reducing storage for data center, the need to implement the data deduplication mechanism and keeping backup of those data in data center is becoming more important. In the case of reducing redundant data from the various servers in data center, the huge amount of data become the biggest challenge. As the data deduplication mechanism not only it can reduce the size of data to be stored or to be transmit but also it can give the backup and recovery features. This paper proposed the efficient data deduplication mechanism called EntroDeduper, which is the hybrid of entropy, secure hash algorithm and indexing. The input files are divided into blocks using file separator and the redundant block is removed by applying the entropy gain theory, which is used to get not only the significant reduction rate but also the full block coverage for reconstruction.