Learned Adapters Are Better Than Manually Designed Adapters
Yuming Zhang, Peng Wang, Ming Tan, Wei Zhu · 2023
Recently, a series of works have looked into further improving the adapter-based tuning by manually designing better adapter architectures.Understandably, these manually designed solutions are sub-optimal.In this work, we propose the Learned Adapter framework to automatically learn the optimal adapter architectures for better task adaptation of pre-trained models (PTMs).First, we construct a unified search space for adapter architecture designs.In terms of the optimization method on the search space, we propose a simple-yet-effective method, GDNAS, for better architecture optimization.Extensive experiments show that our Learned Adapter framework can outperform the previous parameter-efficient tuning (PETuning) baselines while tuning comparable or fewer parameters.Moreover: (a) the learned adapter architectures are explainable and transferable across tasks.(b) We demonstrate that our architecture search space design is valid.