Reducer capacity and communication cost in MapReduce algorithms design
Foto Afrati, Shlomi Dolev, Ephraim Korach, Shantanu Sharma, Jeffrey David Ullman · 2015
An important parameter to be considered in MapReduce algorithms is the "reducer capacity," is introduced here for the first time. The reducer capacity is an upper bound on the sum of the sizes of the inputs that are assigned to the reducer. We consider, for the first time, the different sizes of the inputs, which are sent to the reducers. Another significant parameter in a MapReduce job is "communication cost" -- the total amount of data transferred -- between the map and reduce phases. The communication cost can be minimized by minimizing the number of copies of inputs sent to the reducers.