Effective Masked Language Modeling for Temporal Commonsense Reasoning
Mayuko Kimura, Lis Kanashiro Pereira, Ichiro Kobayashi · 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · 2022
We propose an enhanced language model for temporal commonsense inference. Through a specially designed masked language modeling task, we equip our model with temporal commonsense knowledge. Specifically, we propose different masking strategies for temporal commonsense reasoning. We evaluate our model on a challenging temporal commonsense reasoning dataset. Our experiments show that the method that takes into account the structure of the target dataset, in addition to time-related features, improved the accuracy by approximately 4.8% compared to the standard fine-tuning approach. This confirms that tailoring a pre-trained language model to the domain of a target task is effective.