PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs
Dan Peng, Zhihui Fu, Jun Wang · 2024
Recent advancements in large language models (LLMs) have indeed showcased their impressive capabilities.On mobile devices, the wealth of valuable, non-public data generated daily holds great promise for locally fine-tuning personalized LLMs, while maintaining privacy through on-device processing.However, the constraints of mobile device resources pose challenges to direct on-device LLM fine-tuning, mainly due to the memory-intensive nature of derivative-based optimization required for saving gradients and optimizer states.To tackle this, we propose employing derivative-free optimization techniques to enable on-device finetuning of LLM, even on memory-limited mobile devices.Empirical results demonstrate that the RoBERTa-large model and OPT-1.3Bcan be fine-tuned locally on the OPPO Reno 6 smartphone using around 4GB and 6.5GB of memory respectively, using derivative-free optimization techniques.This highlights the feasibility of on-device LLM fine-tuning on mobile devices, paving the way for personalized LLMs on resource-constrained devices while safeguarding data privacy.