Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, Fuli Feng · 2024
The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions.Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction.From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps.We also design diverse prompts to thoroughly evaluate eleven representative LLMs.Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closedsource models GPT-3.5 and Gemini Pro.Notably, calculation error proves the most challenging error type.Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9%.These results reveal potential directions for developing the mathematical reasoning abilities of LLMs.Our code and dataset is available on https://github.com/LittleCirc1e/EIC.Question: James buys 6 t-shirts for 50% off.They each cost $20.How much did he pay?Correct Solution: Step1: Each shirt cost 20*.5=$10with the sale Step 2: So he paid 10*6=$60 Correct Answer: 60 Wrong Solution: Step1: Each shirt cost 20*.5=