Improving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task
Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao · 2023
Numbers have unique characteristics to words.Teaching models to understand numbers in text is an open-ended research question.Instead of discussing the required calculation skills, this paper focuses on a more fundamental topic: understanding numerals.We point out that innumeracy-the inability to handle basic numeral concepts-exists in most pretrained language models (LMs), and we propose a method to solve this issue by exploring the notation of numbers.Further, we discuss whether changing notation and pre-finetuning along with the comparing-number task can improve performance in three benchmark datasets containing quantitative-related tasks.The results of this study indicate that input reframing and the proposed pre-finetuning task is useful for RoBERTa.