In-Context Learning Creates Task Vectors
Roee Hendel, Mor Geva, Amir Globerson · 2023
In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm.However, its underlying mechanism is still not well understood.In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set S to find a best-fitting function f (x) in some hypothesis class.Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single "task vector" calculated from the training set.Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output.We support the above claim via comprehensive experiments across a range of models and tasks. 1