Detection and Localization of Content deduplication using 64-bit architecture of SHA-256
Alka Leekha, Alam N. Shaikh · 2021 IEEE 6th International Conference on Computing, Communication and Automation (ICCCA) · 2021
Management of big data on cloud storage is a big concern now a days. Data is coming on the cloud from various resources like mobile phones, IOT devices, thin clients etc. This overclouded data sometime leads to server overflow. Handling of this exploded data is a great challenge, As according to EMC surves 75% of data on remote servers is duplicate data. Efficient methodology is required for detecting and localizing the duplicate contents for proper utilization of the storage space and to reduce the cost of buying extra storage. Deduplication can be done both at file level as well as block level and different methods are used for eliminating duplicate chunk. In this paper, we proposed a methodology for localization of content deduplication on variety of data such as text files, audio,images and videos. As size of image and video files are very large generation of chunk and assigning index value to the chunks can cause collision so rather than using already existing algorithms,we used 64-bit architecture of SHA-256 which is more scalable than the existing SHA-256.