Providing caches for reduce tasks in a MapReduce cloud
Tzu-Chi Huang, Kuo-Chih Chu, Jhe-Ru Chen, Xue-Yan Zeng, Ce-Kuen Shieh · 2016
A MapReduce cloud is the key to the success of cloud computing nowadays by the capability of processing large datasets simultaneously on nodes in a cloud. However, a MapReduce cloud may waste many CPU resources to frequently process similar intermediate data in its Reduce tasks because specific intermediate data is always moved to specific Slave nodes. A MapReduce cloud can utilize the proposed idea of supporting the cache mechanism for Reduce tasks to avoid unnecessary computation. In experiments, a MapReduce cloud is proved to get great performance improvement from the help of the cache mechanism when running CPU-intensive applications. Accordingly, a MapReduce cloud can be justified to have the extension of the cache mechanism proposed in this paper.