MemUsing: Dynamic, Efficient memory utilization in compute nodes for HPC memory-based storage systems
Proceedings of 2017 the 7th International Workshop on Computer Science and Engineering · 2017
At present, High Performance Computer (HPC) is developing rapidly from Pflops to Escale.Traditional HDD-based storage systems have become a serious bottleneck hindering the development.It is promising to leverage memory resources in compute nodes to build high-performance storage systems.A memory-based data server can provide approximately 5GB/s read or write bandwidth, which is about 10 times faster than the HDD-based OST in Lustre file system.We propose MemUsing, an efficient method to leverage idle memory among compute nodes.All compute nodes are divided into two partitions, which are full-memory-partition (FMP) and small-memory-partition (SMP).MemUsing introduces a dynamic memory management method to borrow/return memory resources from/to compute nodes in SMP based on the memory requirements of user processes and avoids user processes to be terminated abnormally due to lack of memory.MemUsing dramatically improves the efficiency of memory utilization and the overall performance of memory-based storage systems.