CodePlugin: plugging deduplication into erasure coding for cloud storage
Mengbai Xiao, Mohammed Anowarul Hassan, Weijun Xiao, Qi Wei, Songqing Chen · IEEE International Conference on Cloud Computing Technology and Science · 2015
Cloud storage systems play a key role in many cloud services. To tolerate multiple simultaneous disk failures and reduce the storage overhead, today cloud storage systems often employ erasure coding schemes. To simplify implementations, existing systems, such as Microsoft Azure and EMC Atmos, only support file appending operations. However, this feature leads to a non-trivial and increasing portion of redundant data on cloud storage systems. To reduce the data redundancy due to file updates by users so as to reduce the corresponding encoding and storage cost, in this work, we investigate how to efficiently integrate the inline deduplication capability into the general context of the Reed-Solomon (RS) code. For this purpose, we present our initial design of Code-Plugin. Basically, CodePlugin introduces some preprocessing steps before the normal encoding. In these pre-processing steps, the data duplications are identified and properly shuffled so that the redundant blocks do not have to be encoded. CodePlugin is applicable to any existing coding scheme and our preliminary experimental results show that CodePlugin can effectively improve the encoding throughput (by ∼ 20%) and reduce the storage cost (by ∼ 17.4%).