FedConsDG: Divergence-Aware Federated Distillation via Consensus-Diversity Generative Learning
Shaofan Li, Mingjun Dai, Ruofan Jin · IEEE Transactions on Network Science and Engineering · 2025
Federated learning (FL) is an emerging privacy-preserving distributed learning paradigm that enables multiple clients to collaboratively train a shared model by exchanging model parameters instead of raw data. However, data heterogeneity among clients often leads to significant performance degradation. Existing approaches primarily address this challenge by constraining local model updates to align the global model. Nevertheless, they often overlook the fact that direct aggregation of local models into a single global model can inherently introduce severe performance bottlenecks due to client divergence. To address this issue, we propose FedConsDG: Divergence-Aware Federated Distillation via Consensus-Diversity Generative Learning. Instead of simply aggregating local models, our method updates the global model on the server side via knowledge distillation from local models. Specifically, we deploy a generator on the server and propose a Consensus-Diversity Generative Learning, which encourages the generator to generate hard samples by maximizing the prediction variance among local models. These synthesized samples are then used to fine-tune the global model. To further mitigate catastrophic forgetting caused by data heterogeneity, we introduce a divergence-aware exponential moving average (EMA) mechanism to update the parameters of the generator during the distillation stage. Unlike traditional knowledge distillation techniques, the proposed framework eliminates the need for proxy data on the server, thereby offering stronger privacy guarantees. Extensive experiments on multiple heterogeneous datasets demonstrate the effectiveness of the proposed method.