User Inference Attacks on Large Language Models
Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher A. Choquette-Choo, Zheng Xu · 2024
Text written by humans makes up the vast majority of the data used to pre-train and finetune large language models (LLMs).Many sources of this data-like code, forum posts, personal websites, and books-are easily attributed to one or a few "users".In this paper, we ask if it is possible to infer if any of a user's data was used to train an LLM.Not only would this constitute a breach of privacy, but it would also enable users to detect when their data was used for training.We develop the first effective attacks for user inferenceat times, with near-perfect success-against LLMs.Our attacks are easy to employ, requiring only black-box access to an LLM and a few samples from the user, which need not be the ones that were trained on.We find, both theoretically and empirically, that certain properties make users more susceptible to user inference: being an outlier, having highly correlated examples, and contributing a larger fraction of data.Based on these findings, we identify several methods for mitigating user inference including training with example-level differential privacy, removing within-user duplicate examples, and reducing a user's contribution to the training data.Though these provide partial mitigation, our work highlights the need to develop methods to fully protect LLMs from user inference.Pre-trained LLM Finetuned