Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions
Jad Kabbara, Jackie Chi Kit Cheung · 2023
We explore the connection between presupposition, discourse and sarcasm and propose to leverage that connection in a transfer learning scenario with the goal of improving the performance of NLI models on cases involving presupposition.We exploit advances in training transformer-based models that show that pre-finetuning--i.e., finetuning the model on an additional task or dataset before the actual finetuning phase--can help these models, in some cases, achieve a higher performance on a given downstream task.Building on those advances and that aforementioned connection, we propose pre-finetuning NLI models on carefully chosen tasks in an attempt to improve their performance on NLI cases involving presupposition.We notice that, indeed, pre-finetuning on those tasks leads to performance improvements.Furthermore, we run several diagnostic tests to understand whether these gains are merely a byproduct of additional training data.The results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks plays a role in the performance improvements.