A cost-aware object management method for in-memory computing frameworks
Chin-Hsien Wu, Chien-Wei Chen, Kai-Chun Wang · 2018
For in-memory computing frameworks such as Apache Spark [5, 6], objects (i.e., the intermediated data) can be accommodated in the main memory for speeding up the execution process. In this paper, we propose a cost-aware object management method for in-memory computing frameworks. When the main memory space of any worker node is not enough to accommodate the new computed or the retrieved object, we first pick appreciate objects which are already accommodated in the main memory as candidates for eviction and then evict objects with the minimal sum of the creation cost and the maximum sum of the occupied main memory space. According to the experimental results, we can achieve the goal under the 80/20 and 50/50 principles.