Can Federated Learning Clients be Lightweight? A Plug-and-Play Symmetric Conversion Module
Jialiang Liu, Huawei Huang, Chun Wang, Ruixin Li, Ting Car, Qinglin Yang, Zibin Zheng · 2024
Non-identically distributed (Non-IID) data is a ma-jor challenge in federated learning (FL). Although many related studies have proposed methods to improve FL model performance, they often incur significant resource consumption. These studies typically save gradient states for training correction, some requiring clients to synchronize these states. Given that clients' extra gradient states could be substantial, even several times larger than the model's size, maintaining and synchronizing such large-size gradient states consume considerable memory and communication resources. This paper rigorously explores a substantial reduction in Non-IID methods' resource consumption on clients by reconstructing Non-IID methods' local corrections on the server. A crucial insight driving this reconstruction is to ensure symmetrical execution time for corrections. Motivated by this principle, we introduce Fleet, a lightweight FL framework. Fleet's server performs a two-stage symmetric gradient correction, while clients perform original gradient descents. Experimen-tal results demonstrate Fleet's superior performance over state-of-the-art methods, with resource consumption comparable to lightweight FedAvg on clients. Especially, Fleet excels in training deep models using large datasets. The experimental findings also support Fleet's dynamic scheduling as a plug-and-play module, showcasing its practical potential in real-world applications.