A New Scheme for Cache Optimization Based on Cluster Computing Framework Spark
Kun Wang, Ke Zhang, Chengxue Gao · 2015
Nowadays people have paid more attention to use the memory to do data transmission of the parallel computing framework in the iterative computations. Compared to the traditional disk and network, using the memory can not only reduce the transmission time of data, but also improve running speed. Spark memory computational framework is the most popular data processing model currently and also has better fault tolerance. This paper mainly analyze the Spark's usage behavior of memory, aims at improving the execution order of RDD's operator action, and compute the weights of RDDs in the task. Finally, the paper proposes an ASRW algorithm, which has important reference value to improve memory cache resource utilization rate and improve the running efficiency of the program.