Zeroshot Crosslingual Transfer of a Gloss Language Model for Semantic Change Detection
Maxim Rachinskiy, Nikolay Arefyev · Computational Linguistics and Intellectual Technologies · 2021
Consulting word definitions from a dictionary is a familiar way for a human to find out which senses a particular word has.We hypothesize that a system that can select a proper definition for a particular word occurrence can also naturally solve Semantic Change Detection (SCD) task.To verify our hypothesis, we followed an approach previously proposed for Word Sense Disambiguation (WSD) and trained a system that embeds word definitions and word occurrences into the same vector space.In this space, the embedding of the most appropriate definition has the largest dot product with a contextualized word embedding.The system is trained on an English WSD corpus.To make it work for the Russian language, we replaced BERT with the multilingual XLMR language model and exploited its zeroshot crosslingual transferability.Despite not finetuning the encoder model on any Russian data, this system achieves the second place in the competition, and likely works for any of one hundred other languages XLMR was pretrained on, though the performance may vary.We then measure the impact of such WSD pretraining and show that this procedure is crucial for our results.Since our model was trained to choose a proper definition for a word, we propose an algorithm for the interpretation and visualization of the semantic changes through time.By employing additional labeled data in Russian and training a simple regression model, that converts the distances between output contextualized embeddings into more humanlike scores of sense similarity between word occurrences, we further improve our results and achieve the first place in the competition.