FlashMatrix: Parallel, Scalable Data Analysis with Generalized Matrix Operations using Commodity SSDs.
Da Zhong Zheng, Disa Mhembere, Joshua T Vogelstein, Carey E. Priebe, Randal C. Burns · arXiv (Cornell University) · 2016
FlashMatrix is a matrix-oriented programming framework for general data analysis with high-level functional programming interface. It scales matrix operations beyond memory capacity by utilizing solid-state drives (SSDs) in non-uniform memory architecture (NUMA). It provides a small number of generalized matrix operations (GenOps) and reimplements a large number of matrix operations in the R framework with GenOps. As such, it executes R code in parallel and out of core automatically. FlashMatrix uses vectorized user-defined functions (VUDF) to reduce the overhead of function calls and fuses matrix operations to reduce data movement between CPU and SSDs. We implement multiple machine learning algorithms in R to benchmark the performance of FlashMatrix. The execution of the R implementations in FlashMatrix has performance comparable to optimized C implementations. When scaling beyond memory capacity on a large parallel machine, the out-of-core execution of these R implementations in FlashMatrix has performance comparable to in-memory execution on a billion-scale dataset. Both in-memory and out-of-core execution significantly outperform the in-memory execution of Spark MLlib.