Sanitizing Sentence Embeddings (and Labels) for Local Differential Privacy
Minxin Du, Xiang Yue, Sherman S. M. Chow, Huan Sun · 2023
Differentially private (DP) learning, notably DP stochastic gradient descent (DP-SGD), has limited applicability in fine-tuning gigantic pre-trained language models (LMs) for natural language processing tasks. The culprit is the perturbation of gradients (as gigantic as entire models), leading to significant efficiency and accuracy drops.