Methodologies and Advances in Fine-Tuning Techniques for Large Language Models
Zhu Huang, Hongying Luo, Yizhu Wen, Tianyang Wang, Ziqian Bi, Junhao Song, Ming Liu, Xinyuan Song, Junfeng Hao, Chia Xin Liang · 2025
This paper provides a comprehensive overview of fine-tuning techniques for Large Language Models (LLMs), a critical component in advancing natural language processing. It synthesizes recent progress in instruction fine-tuning, multitask learning, federated learning, and pedagogical alignment, highlighting their effectiveness, challenges, and potential applications. The survey also delves into technical aspects such as model specialization, evaluation metrics, and heterogeneous client management in federated settings. Moreover, it addresses open issues including societal implications and ethical considerations. Key findings emphasize the significance of tailored fine-tuning approaches for enhancing LLM adaptability, the value of comprehensive evaluation protocols, and the impact of informed instruction selection. This work contributes to the field by analyzing current trends, summarizing technical advancements, and outlining future directions. It aims to support researchers and practitioners in navigating the complexities of LLM fine-tuning and in developing more robust, generalizable, and responsible NLP systems across diverse domains.