D$^{3}$ : A Dynamic Dual-Phase Deduplication Framework for Distributed Primary Storage
Jianwei Yin, Yan Jun Tang, Shuiguang Deng, Ying Kui Li, Albert Y. Zomaya · IEEE Transactions on Computers · 2017
Deploying deduplication for distributed primary storage is a sophisticated and challenging task, considering that the demands of low read/write latency, stable read/write performance, and efficient space saving are all of paramount importance. Unfortunately, existing schemes cannot present a satisfactory solution for the aforementioned requirements simultaneously. In this article, we propose D$^{3}$, a dynamic dual-phase deduplication framework for distributed primary storage. Several major innovations are established in D$^{3}$. First, we formulate a deduplication-oriented taxonomy calledDedup-Type, to group data with similar deduplication-related characteristics into larger categories. It serves as coarse-grained filter and one of the prioritizing references in D$^{3}$. Second, D$^{3}$is a dual-phase framework—inline-phase and offline-phase deduplication processes work in concert with each other. Third, D$^{3}$operates in a dynamic manner. We design two critical mechanisms:context-aware threshold adjustment(CTA) for local inline-phase deduplication, anddeferred priority-based enforcement(DPE) for global offline-phase deduplication. The CTA mechanism enables selective deduplication under a periodically updated threshold. Data skipped during the inline phase is regarded as a candidate for offline phase, and is handled in a prioritized order under the governance of DPE mechanism. Evaluation results demonstrate that, compared with conventional inline and offline deduplication schemes, D$^{3}$achieves more efficient and stabler read/write performance with competitive space saving.