Big Models, Small Tweaks: The Rise of Parameter-Efficient Fine-Tuning
Xia Yang, Zhihao Chang, Shufen Zhihao · 2025
Foundation models have revolutionized artificial intelligence (AI) by enabling state-of-the-art performance across a wide range of tasks. However, fine-tuning these massive models presents significant computational and storage challenges, making it impractical for many real-world applications. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a promising solution, allowing efficient adaptation of foundation models with minimal modifications. This survey provides a comprehensive review of PEFT techniques, including Adapters, Low-Rank Adaptation (LoRA), Prefix Tuning, Prompt Tuning, and BitFit. We analyze their theoretical foundations, empirical performance across natural language processing (NLP), computer vision (CV), and multimodal tasks, and their real-world applications in domains such as healthcare, robotics, and scientific computing. Additionally, we discuss the key challenges in PEFT, including optimization stability, generalization, robustness, and scalability to increasingly large models. We highlight open research directions such as hybrid PEFT strategies, integration with federated learning, and applications in continual learning. By synthesizing recent advancements, this survey aims to provide a structured understanding of PEFT and its role in democratizing access to foundation models, paving the way for more efficient and scalable AI systems.