On Implementation of Parallel Map and Shuffle Phases for Coded Distributed Computing
Haoyu Tu, Kai Yuan, Shuai Ma, Youlong Wu · IEEE Communications Letters · 2021
In this letter, we consider the MapReduce-type distributed system and jointly design the Map and Shuffle phases of the coded distributed computing (CDC) scheme proposed by Li,et al.By delicately setting the order of file mapping and data shuffling, we implement the Map and Shuffle phases in parallel, and thus significantly reduce the overall task latency (time cost at the Map, Shuffle, Reduce operations and codes generation). Furthermore, our scheme enables all nodes to remove their auxiliary intermediate values once these values are no longer useful in the Shuffle phase, leading to great reduction in the storage cost of the system.