Partial Training Mechanism to Handle the Impact of Stragglers in Federated Learning with Heterogeneous Clients
Bruno S. Martins, Allan Mariano De Souza, Denis Lima do Rosário, Carlos A. Astudillo, Eduardo Cerqueira, Leandro Aparecido Villas · 2024
Federated Learning (FL) allows distributed devices, known as clients, to train Machine Learning (ML) models collaboratively without sharing sensitive data. A characteristic of FL for mobile and IoT environments is system heterogeneity among clients, which can vary from low-end devices with constrained communication and computing resources to powerful devices with high-speed network access and dedicated GPUs. As the server must wait for all the clients to communicate their updates, slow clients (a.k.a. stragglers) will significantly increase the training time. To tackle this problem, we propose FedPulse, a Partial Training (PT) based mechanism to mitigate the effect of stragglers in FL. The idea is to reduce the training time by dynamically allocating smaller submodels to resource-constrained clients. Experimental results on famous classification datasets show that the proposed solution outperforms other submodel allocation mechanisms and reduces the training time by up to 58% with an accuracy loss of less than 1% when compared to FedAvg.