Privacy Leakage from Logits Attack and its Defense in Federated Distillation
Danyang Xiao, Diying Yang, Jialun Li, Xu Chen, Weigang Wu · 2024
Federated Distillation (FD), a popular variant of Federated Learning (FL), has attracted researchers' attention due to its ability to support heterogeneous model training. Generally, FD allows clients to upload logits associated with public datasets for knowledge transfer, yet logits may pose privacy risks. In this study, we provide the first demonstration of the impact of privacy risks caused by logits. Specifically, we design a data reconstruction attack against logits named L-Attack which can reveal sensitive information about the target client without access to the target model. Via the zeroth-order optimization technique, L-Attack involves training a server-side generator that unveils certain features of private data owned by the target client. To defend against L-Attack, we propose a label aggregation-based FD algorithm called LabelAvg which allows clients to upload predicted hard labels for knowledge transfer instead of logits. Due to the insufficient information in labels for distillation, LabelAvg provides a voting-based label smoothing mechanism that enables the server to construct smooth labels from received labels. The generated smooth labels which stand for the consensus among all clients, indicate the approximate probability distribution. Thus, these smoothed labels bear a striking similarity to logits and can be used for distillation. Analysis and experimental results prove LabelAvg is superior to baselines in terms of accuracy, privacy, and communication data volume.