TOU: A Truncated-factorized reduction for a lightweight fine-tuning method
Phuc Binh Nguyen, Koji Zettsu · 2025
The full fine-tuning pre-trained models represents an effective approach. But it typically requires the updating of all model parameters, which can result in in computational and memory costs. To address this challenge, we present a methodology called Truncated-factorized reduction (TOU), a methodology that leverages Truncated Singular Value Decomposition (TSVD) to factorize each pre-trained weight matrix into two smaller matrices. One factor is strategically frozen, preserving core knowledge from the pre-train phase, while the other factor is subjected to standard fine-tuning procedures. After fine-tuning, the original weight matrices are reconstructed by matrix multiplication of the two factors, effectively updating the full weights. This method significantly reduces the number of trainable parameters during fine-tuning, leading to enhanced efficiency in terms of computation and memory. Furthermore, by introducing a novelty efficiency parameter, we provide a mechanism to control the trade-off between fine-tuning efficiency and model adaptability by adjusting the truncation level in TSVD. We hypothesize and aim to demonstrate that TOU offers a compelling strategy for efficient and effective full fine-tuning, enabling faster adaptation of large models to new tasks while maintaining performance compared to traditional full fine-tuning. Experiments on the Vision Transformer model show that TOU achieves a 70% reduction in trainable parameters while maintaining (accuracy drops < 1%), reduces 65% in terms of training time, and 27% in terms of memory usage comparable performance to full weight fine-tuning. Furthermore, TOU delivers better performance than LoRA in terms of training speed and memory usage when applied to fine-tune GPT-2M with comparable performance. This approach holds promise in accelerating the fine-tuning model by reducing the computational burden associated with their downstream application.