Meta-Learning Online Adaptation of Language Models
Nathan Hu, Eric Mitchell, Christopher D. Manning, Chelsea Finn · 2023
Large language models encode impressively broad world knowledge in their parameters.However, the knowledge in static language models falls out of date, limiting the model's effective "shelf life."While online fine-tuning can reduce this degradation, we find that naively fine-tuning on a stream of documents leads to a low level of information uptake.We hypothesize that online fine-tuning does not sufficiently attend to important information.That is, the gradient signal from important tokens representing factual information is drowned out by the gradient from inherently noisy tokens, suggesting that a dynamic, context-aware learning rate may be beneficial.We therefore propose learning which tokens to upweight.We meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-ofdate base question-answering model's ability to answer questions about a document after a single weighted gradient step.We call this approach Context-aware Meta-learned Loss Scaling (CaMeLS).Across three different distributions of documents, our experiments find that CaMeLS provides substantially improved information uptake on streams of thousands of documents compared with standard fine-tuning and baseline heuristics for reweighting token losses.