Convergence Analysis for Federated Dropout
Sijing Xie, Dingzhu Wen, Xiaonan Liu, Changsheng You, Tharmalingam Ratnarajah, Kaibin Huang · 2024
Federated dropout on the weight is an efficient technique to overcome both communication and computation bottlenecks for deploying federated learning at the network edge. However, the theoretical analysis for Federated Dropout is still lacking in the literature, due to the challenge arising from the gradient bias. To address this issue, by using the Taylor expansion method, we mathematically show that the gradient vector with dropout can be approximated as an unbiased estimation of that without dropout; while its gradient variance increases with a scaling factor of γ/(1 − γ), with γ ∈ [0,θ) denoting the dropout rate and θ being the maximum dropout rate ensuring the loss function reduction. Based on the above approximation, we provide the loss function analysis for Federated Dropout. Specifically, it is shown that a larger dropout rate of each device leads to a slower convergence rate. Finally, numerical results are provided to verify the effects of dropout rate on convergence in both underfitting and overfitting scenarios.