Application of speculative execution technology for communication optimization in HPMR
Qilong Zheng · 2010
HPMR (high performance MapReduce)system is a high performance computing software supporting platform.It is common for HPMR to do many rounds of the Map-Reduce process in high performance computing.The middle communication process consists of KV routing and KV transfer.KV transfer is carried out with the help of the KV table generated by KV routing.The notable feature of communication in HPMR program is that each round of KV transfer adopts the KV table of the last round with high probability.In view of to this characteristic,speculation execution technology is introduced into the communication model of HPMR,which reduces the number of KV routings and greatly enhances communication performance of HPMR.