A Distributed Cache Framework on Massively Parallel Computing Architecture
Proceedings of 2017 the 7th International Workshop on Computer Science and Engineering · 2017
Current high performance computers (HPC) mostly adopt massively parallel computing (MPP) architecture, which uses parallel file system to accommodate all the data, instead of having local storage located on the compute node side.MPP architecture prevents large scale applications dumping terabytes of temporal or intermediate data into local storage and causes severe I/O bottleneck.In this paper, we proposed a distributed cache framework to merge scattered memory spaces of up to thousands of compute nodes into a unified cache pool, so that data-intensive applications can store their intermediate data into those compute nodes of light I/O burden.We evaluate the framework with IOR and a realistic benchmark BTIO on a HPC system TH-1A, which indicates that our approach can bring significant performance boost to data-intensive applications.