TRAM: Benchmarking Temporal Reasoning for Large Language Models
Yuqing Wang, Yun Zhao · 2024
Reasoning about time is essential for understanding the nuances of events described in natural language.Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies.In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs).We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domainspecific models.Our findings indicate that the best-performing model lags significantly behind human performance.It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.Our data and code are available at https:// github.com/EternityYW/TRAM-Benchmark.Q: It is also a love story , between Ace and Tobio, a trans woman.How often do they break up? A. Once B. Always C. Once per weekFrequency (Commonsense) Q: A historic event is documented to have happened 'before you know it'.When did it take place? A. The next day B. Without hesitation C. Before long Ambiguity Resolution (Interpretation) Q: She noticed that all the wall clocks in the store were set to ten past ten.What's the more plausible CAUSE? A. It is a common display setting for clocks and watches.B. B. It was ten minutes past ten at that moment.Temporal Causality (Cause) Q: I woke up so late this morning.I was panicked when I saw what time it was.I had to be at work on time.I threw myself together quickly.Which of the two endings is the most plausible correct ending to the story? A. I was able to get a job at a local restaurant.B.