Towards a Language Model for Temporal Commonsense Reasoning
Mayuko Kimura, Lis Kanashiro Pereira, Ichiro Kobayashi · Student Research Workshop .../Proceedings of the Student Research Workshop ... · 2021
Temporal commonsense reasoning is a challenging task as it requires temporal knowledge usually not explicitly stated in text.In this work, we propose an ensemble model for temporal commonsense reasoning.Our model relies on pre-trained contextual representations from transformer-based language models (i.e., BERT), and on a variety of training methods for enhancing model generalization: 1) multistep fine-tuning using carefully selected auxiliary tasks and datasets, and 2) a specifically designed temporal task-adaptive pre-trainig task aimed to capture temporal commonsense knowledge.Our model greatly outperforms the standard fine-tuning approach and strong baselines on the MC-TACO dataset.