NLMs: Augmenting Negation in Language Models
Rituraj Singh, Rahul Kumar, Vivek Sridhar · 2023
Negation is the fundamental component in a natural language that reverses the semantic meaning of a sentence.It plays an extremely important role across a wide range of applications, yet they are under-represented in pretrained language models (LMs), resulting often in wrong inferences.In this work, we try to improve the underlying understanding of the negation in the pre-trained LMs.To augment negation understanding, we propose a language model objective with a weighted cross-entropy loss and elastic weight consolidation regularization.For negated augmented models, we reduce the mean top 1 error rate for BERTbase to l.1%, BERT-large to 0.78%, RoBERTabase to 3.74%, RoBERTa-large to 0.01% on the negated LAMA dataset that outperform the existing negation models.It minimizes the mean error rate by a margin of 8% and 6% for original BERT and RoBERTa models.We also provide empirical evidences that negated augmented models outperforms the classical models on original as well as negation benchmarks on natural language inference tasks.