KnFu: Effective Knowledge Fusion
Seyed Jamal Seyedmohammadi, S. Kawa Atapour, Jamshid Abouei, Arash Mohammadi · 2024
Federated Learning (FL) is a decentralized approach that allows for collaborative training of Machine Learning (ML) models across multiple local nodes, ensuring data privacy and security while leveraging diverse datasets. Conventional FL, however, is susceptible to gradient inversion attacks, restrictively enforces a uniform architecture on local models, and suffers from model heterogeneity (model drift) due to non-IID local datasets. To mitigate some of these challenges, the new paradigm of Federated Knowledge Distillation (FKD) has emerged. FKD is developed based on the concept of Knowledge Distillation (KD), which involves extraction and transfer of a large and well-trained teacher model’s knowledge to lightweight student models. FKD, however, still faces the model drift issue. Intuitively speaking, not all knowledge is universally beneficial due to the inherent diversity of data among local nodes. This calls for innovative mechanisms to evaluate the relevance and effectiveness of each client’s knowledge for others, to prevent propagation of adverse knowledge. In this context, the paper proposes Effective Knowledge Fusion (KnFu) algorithm that evaluates knowledge of local models to only fuse semantic neighbors’ effective knowledge for each client. The KnFu is a personalized effective knowledge fusion scheme for each client, that analyzes effectiveness of different local models’ knowledge prior to the aggregation phase. In this context, closeness of clients’ knowledge is measured by estimating the class distributions of local datasets based on the transmitted localized knowledge. Comprehensive experiments were performed on MNIST and CIFAR10 datasets. KnFu outperforms its FL-based, FKD-based, and local training baselines in scenarios where the clients have small datasets with intermediate degree of heterogeneity. KnFu’s source code is accessible throught the following link: https://github.com/jamal94sm/KnFu-Effective-Knowledge-Fusion.git.