Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation
Xiaohang Tang, Yi Xin Zhou, Danushka Bollegala · 2023
Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words.We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using timesensitive templates.Given two snapshots C 1 and C 2 of a corpus taken respectively at two distinct timestamps T 1 and T 2 , we first propose an unsupervised method to select (a) pivot terms related to both C 1 and C 2 , and (b) anchor terms that are associated with a specific pivot term in each individual snapshot.We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms.Moreover, we propose an automatic method to learn time-sensitive templates from C 1 and C 2 , without requiring any human supervision.Next, we use the generated prompts to adapt a pretrained MLM to T 2 by fine-tuning using those prompts.Multiple experiments show that our proposed method reduces the perplexity of test sentences in C 2 , outperforming the current state-of-the-art.