Theoretical Limits of Feedback Alignment in Preference-Based Fine-Tuning of AI Models
Zhenyu Gao · 2025
Feedback alignment (FA) has emerged as an alternative to backpropagation for training deep networks by using fixed random feedback weights. While FA shows promise in supervised tasks, its extension to preference-based fine-tuning (PFT) of large language models—which relies on human or learned preference signals—remains underexplored. In this work, we analyze theoretical limitations of FA applied to PFT objectives. We derive error propagation bounds, characterize convergence conditions for paired-FA updates, and quantify the impact of preference noise and feedback mismatch on fine-tuning stability. By integrating recent advances in meta-reinforcement learning and prompt compression, we highlight trade-offs between feedback complexity and fine-tuning efficiency, offering practical guidelines for hybrid FA-backprop architectures in large-scale preference optimization.