An Out-of-Core Dataflow Middleware Applied to Spare Linear Algebra
Zheng Zhou, Xianbin Xu · International Journal of Digital Content Technology and its Applications · 2013
Many applications in scientific computing require a tremendous amount of data, running on thousands of computing nodes so as to ensure all the data fits in the main memory of the distributed machine. However, for some irregular applications, like sparse linear solvers, running on thousands of nodes typically do not achieve high efficiency, leading to a significant waste of computing time. Out-ofcore computing allows using hard drives to store the state of the application. In this paper, we propose to build a task-based out-of-core programming model based on the filter stream programming model. Coupling a distributed storage system with a data-aware scheduler allows achieving communication and computation overlapping as well as a reutilization. Experiments on an in-house cluster equipped with regular hard drive indicates that using a single SSD drive per node should be enough to reach equivalent computation time usage between in-core and out-of-core computations, while a RAID SSD configuration will achieve higher efficiency in out-of-core mode.