De²Dup: Extended Deduplication for Multi-Tenant Databases
Alexander Krause, Jannis Kowalick, Johannes Pietrzyk, Dirk Habich, Wolfgang Lehner · 2025
Content-based page sharing (de-duplication) is a heavily used technique to improve memory efficiency in virtualized systems by identifying and merging identical pages.For many years now, the Linux kernel has offered this de-duplication technique via the Kernel Samepage Merging (KSM) feature.Although KSM in general works well, it is not used in multi-tenant database systems even though multiple tenants often manage similar data.One reason is that pages must be binary identical, which is a severe restriction.Secondly, KSM is seemingly scheduled as a single-threaded process by the OS, independently of the database workload, which further limits its applicability for in-memory systems with terabytes of main memory.To overcome that, we propose an extended de-duplication mechanism called De 2 Dup for memory-centric multi-tenant database engines.De 2 Dup extends de-duplication with a delta mechanism to significantly boost the application, especially when pages are not binary identical.Moreover, our De 2 Dup mechanism allows to steer the search for duplicates and has low overhead as we are able to offload the complete execution to a modern on-chip accelerator for memory operations in an asynchronous manner on recent Intel server processors.In addition, De 2 Dup offers an efficient way for on-the-fly tenant-aware data reconstruction during scan operations.