Coded Distributed Computing Schemes with Fewer Files
Jing Jiang, Yuqing Xiao · 2023
Coded distributed computing (CDC) is an efficient method to reduce the communication load in general distributed computing frameworks such as MapReduce. In these systems, one usually needs to split the data set into disjoint files to complete a computational task. Li et al. provided some CDC schemes achieving optimal communication load. However, as the number of computing nodes increases, the number of files grows too fast to be applied in practice. In this paper, we first propose a new construction of CDC schemes. Then we show that the communication load in the new scheme is slightly larger than that of the scheme obtained by Li et al, while the number of files in the new scheme is much smaller than that of the scheme obtained by Li et al.