Memory management for billions of small objects in a distributed in-memory storage
Florian Klein, Kevin Beineke, Michael Schöttner · 2014
Large-scale interactive applications and online analytic processing on graphs require fast data access to huge sets of small data objects. DXRAM addresses these challenges by keeping all data always in memory of potentially many nodes aggregated in a data center. In this paper we focus on the efficient memory management and mapping of global IDs to local memory addresses, which is not trivial as each node may store up to one billion of small data objects (16-64 byte) in its local memory. We present an efficient paging-like translation scheme for global IDs and a memory management optimized for many small data objects. The latter includes an efficient incremental defragmentation supporting changing allocation granularities for dynamic data. Our evaluations show that the proposed memory management approach has only a 4-5% overhead compared to state of the art memory allocators with around 20% and the paging-like mapping of globals IDs is faster and more efficient than hash-table based approaches. Furthermore, we compare memory overhead and read performance of DXRAM with RAMCloud.