AMP: Multi-Task Transfer Learning via Leveraging Attention Mechanism on Task Embeddings
Yangyang Yu, Keru Wang · International Journal of Pattern Recognition and Artificial Intelligence · 2025
The attention mechanism has been successfully used in a sequence consisted of a series of word embeddings to improve the representation of the sequence. Inspired by this, we leverage the attention mechanism on a set of tasks to implement a multi-task transfer learning method called AMP (Attentions between Multiple Prompts). First, we encode a task into a prompt as task representation called task embedding. Second, we learn an attention component on all task embeddings to generate a combined prompt for each task, which is an attention-weighted sum of task embeddings. Each combined prompt incorporates the knowledge of all tasks. The word embedding is a vector, but the task embedding is a 2D matrix. The attention mechanism can be exploited on a set of vectors rather than on a set of matrices. The prior methods employ pooling or flattened method to transform the matrix to the vector for computing the attentions between matrices. We propose a method called DAM (Direct Attention Mechanism) which can compute attentions between matrices directly without transforming. DAM method can more exactly compute the attentions between matrices. Wide experiments demonstrate that AMP outperforms prompt-tuning method and prior prompt transfer methods.