Learning to Rank for Plausible Plausibility
Zhongyang Li, Tongfei Chen, Benjamin Van Durme · 2019
Researchers illustrate improvements in contextual encoding strategies via resultant performance on a battery of shared Natural Language Understanding (NLU) tasks.Many of these tasks are of a categorical prediction variety: given a conditioning context (e.g., an NLI premise), provide a label based on an associated prompt (e.g., an NLI hypothesis).The categorical nature of these tasks has led to common use of a cross entropy log-loss objective during training.We suggest this loss is intuitively wrong when applied to plausibility tasks, where the prompt by design is neither categorically entailed nor contradictory given the context.Log-loss naturally drives models to assign scores near 0.0 or 1.0, in contrast to our proposed use of a margin-based loss.Following a discussion of our intuition, we describe a confirmation study based on an extreme, synthetically curated task derived from MultiNLI.We find that a margin-based loss leads to a more plausible model of plausibility.Finally, we illustrate improvements on the Choice Of Plausible Alternative (COPA) task through this change in loss.