DFLStar: A Decentralized Federated Learning Framework with Self-Knowledge Distillation and Participant Selection
Behnaz Soltani, Venus Haghighi, Yipeng Zhou, Quan Z. Sheng, Lina Yao · 2024
Federated learning (FL) is a distributed machine learning paradigm in which clients collaboratively train models in a privacy-preserving manner. While centralized FL (CFL) suffers from single points of failure and performance bottlenecks, decentralized FL (DFL), which depends on inter-client communication, has emerged to eliminate the need of a central entity. However, due to lack of the coordination of a central server, heterogeneous data distribution across clients makes local models in DFL inclined to diverge towards their local objectives, resulting in poor model accuracy. Moreover, each client in DFL needs to communicate with multiple neighbors, yielding a heavy communication load. To tackle these challenges, we propose a novel DFL framework called DFLStar, which can improve DFL from two perspectives. First, to avoid significantly diverging towards local data, DFLStar incorporates self-knowledge distillation to enhance the local model training by assimilating knowledge from the aggregated model. Second, clients in DFLStar identify and only select the most informative neighbors (based on the last layer model similarity) for parameter exchange, thereby minimizing the communication overhead. Our experimental results on two real datasets demonstrate that DFLStar significantly improves both communication overhead and training time compared to traditional DFL algorithms while achieving a specific target accuracy. Furthermore, within a fixed training duration, DFLStar constantly obtains the highest model accuracy compared to the baselines.