ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
Jingyuan Selena She, Christopher E. Potts, Samuel R. Bowman, Atticus Geiger · 2023
A number of recent benchmarks seek to assess how well models handle natural language negation.However, these benchmarks lack the controlled example paradigms that would allow us to infer whether a model had learned how negation morphemes semantically scope.To fill these analytical gaps, we present the Scoped Negation NLI (ScoNe-NLI) benchmark, which contains contrast sets of six examples with up to two negations where either zero, one, or both negative morphemes affect the NLI label.We use ScoNe-NLI to assess fine-tuning and in-context learning strategies.We find that RoBERTa and DeBERTa models solve ScoNe-NLI after many shot fine-tuning.For in-context learning, we test InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning.To better understand this result, we extend ScoNe with ScoNe-NLG, a sentence completion test set that embeds negation reasoning in short narratives.Here, InstructGPT is successful, which reveals the model can correctly reason about negation, but struggles to do so on prompt-adapted NLI examples outside of its core pretraining regime.