Federated Large Language Model: Solutions, Challenges and Future Directions
Jiahui Hu, Dan Wang, Zhibo Wang, Xiaoyi Pang, Huiyu Xu, Ju Ren, Kui Ren · IEEE Wireless Communications · 2024
Large language models (LLMs) have become increasingly popular due to their exceptional performance in various artificial intelligence applications. However, their development often suffers from the scarcity of high-quality data and the extensive requirements for computing resources. These obstacles are even more severe for enterprises in vertical industries, which have limited computer resources but urgently require large-scale models for specific activities. To address these issues, LLMs call for the integration of federated learning (FL), which enables the collaborative learning of a powerful LLM using private data and computing resources from multiple entities. In this article, we present a systematic introduction to the federated large language model (Fed-LLM), a distributed learning of LLM in the FL manner. We first introduce the learning paradigm of Fed-LLM, which is called federated parameter-efficient fine-tuning (Fed-PEFT). Fed-PEFT empowers the collaborative fine-tuning of pretrained LLMs by only involving a small subset of parameters in local LLMs. Specifically, we detail the workflow of Fed-PEFT, and summarize the state-of-the-art solutions in this area. Additionally, we discuss the challenges faced in Fed-LLMs, including efficiency, privacy, and security. Finally, we introduce future directions to facilitate the research of Fed-LLMs and guide coming explorations in this nascent field.