Prompt-based Masked Language Modeling for Numerical Reasoning
Sujit Kumar, Monika Singh, Abhishek Ranjan, Tanveen Tanveen, Sanasam Ranbir Singh · ACM Transactions on Intelligent Systems and Technology · 2025
Headline generation, a crucial task in summarization, aims to summarize an entire article into a concise, single line. Despite the proficiency of sequence-to-sequence encoder-decoder models and transformer-based large language models (LLMs) in text generation and summarization, generating headlines that include numerals representing the numerals in the news body remains a significant challenge. Generating a numeral-aware headline requires the ability of models to solve numerical and mathematical reasoning capabilities to infer relationships between numerals in the news body. Given the challenges in numeral-aware headline generation and numerical reasoning over numerals in news bodies, this study conducts an empirical investigation of LLMs using various strategies, including Pretrained , Few-shot Prompting , and Chain-of-Thought (CoT) Prompting , for numeral-aware headline generation and numerical reasoning for headline generation. Building upon the insights gained from our empirical study on LLMs for numeral-aware headline generation and numerical reasoning, we propose two novel approaches: instruction tuning with LLMs for numeral-aware headline generation and prompt-based masked language modeling for numerical reasoning. We conducted our experiments on the NumHG dataset. We observed that our proposed method outperforms the Pretrained , Few-shot Prompting , and CoT Prompting setups of LLMs, as well as baseline models from the literature, on both numeral-aware headline generation and numerical reasoning tasks. Observations from the experimental results reveal that our proposed Prompt-based Masked Language Modeling significantly improves the performance of small and medium-sized language models on numerical reasoning tasks. We also study the robustness of our proposed models by evaluating their performance in fact-checking numerical claims and performing numerical reasoning for numeral-aware text summarization. Our findings suggest that the proposed Prompt-based Masked Language Modeling approach is also effective for numerical claim verification and numerical reasoning for numeral-aware text summarization.