HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin · 2025
Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses.Do these models truly understand commonsense knowledge, or just memorize expression patterns?To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning.We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants.To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories.We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning.Furthermore, this robustness varies depending on the language in which the LLM is tested.This work establishes a high-quality evaluation benchmark, with extensive experiments offering valuable insights to the community in commonsense reasoning for LLMs.* Corresponding authors.Work done when Xiaoyuan Li and Keqin Bao were intern at Alibaba Group.Question: The lady stands and lifts the weight over her head.Which could be the most possible context for this action?Choices: A. A lady positions herself at the squat rack.She lowers her body before rising steadily.B. A lady stands beside the bench press station.She lies down and lifts the barbell from her chest.C. A lady walks to a barbell.She bends down and grabs the pole.D. A lady approaches the kettlebell set.She swings the weight forcefully between her legs.Answer: C A lady walks to a barbell.She bends down and grabs the pole.The lady stands and lifts the weight over her head.Original question Negative transformation Question: A lady walks to a barbell.She bends down and grabs the pole.The lady Choices: A. swings and lands in her arms.B. pulls the barbell forward.C. pulls a rope attached to the barbell.D. stands and lifts the weight over her head.Answer: D Question: A lady walks to a barbell.She bends down and grabs the pole.The lady will not Choices: A. swing and land in her arms.B. bend her knees and lift the barbell.C. adjust her grip and lift the weight.D. stand and lift the weight