UsingWikidata for Enhancing Compositionality in Pre-trained Language Models
Meriem Beloucif, Mihir Bansal, Chris Biemann, Universität Hamburg, Germany · 2023
One of the many advantages of pre-trained language models (PLMs) such as BERT and RoBERTa is their flexibility and contextual nature.These features give PLMs strong capabilities for representing lexical semantics.However, PLMs seem incapable of capturing highlevel semantics in terms of compositionality.We show that when augmented with the relevant semantic knowledge, PMLs learn to capture a higher degree of lexical compositionality.We annotate a large dataset from Wikidata highlighting a type of semantic inference that is easy for humans to understand but difficult for PLMs, like the correlation between age and date of birth.We use this resource for finetuning DistilBERT, BERT large and RoBERTa.Our results show that the performance of PLMs against the test data continuously improves when augmented with such a rich resource.Our results are corroborated by a consistent improvement over most GLUE benchmark natural language understanding tasks.