Forward-Backward Reasoning in Large Language Models for Mathematical Verification
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, James Tin-Yau Kwok · 2024
Self-Consistency samples diverse reasoning chains with answers and chooses the final answer by majority voting.It is based on forward reasoning and cannot further improve performance by sampling more reasoning chains when saturated.To further boost performance, we introduce backward reasoning to verify candidate answers.Specifically, for mathematical tasks, we mask a number in the question and ask the LLM to answer a backward question created by a simple template, i.e., to predict the masked number when a candidate answer is provided.Instead of using forward or backward reasoning alone, we propose FOBAR to combine FOrward and BAckward Reasoning for verification.Extensive experiments on six standard mathematical data sets and three LLMs show that FOBAR achieves state-of-the-art performance.In particular, FOBAR outperforms Self-Consistency, which uses forward reasoning alone, demonstrating that combining forward and backward reasoning is more accurate in verification.In addition, FOBAR achieves higher accuracy than existing verification methods, showing the effectiveness of the simple template used in backward reasoning and the proposed combination.Backward question (with answer 21): The sum of three consecutive odd numbers is x.What is the smallest of the three numbers?If we know the answer to the above question is 21, what is the value of unknown variable x? (we sample 10 backward chains and all predict x = 69, thus, 10 correct backward chains) Backward question (with answer 23): The sum of three consecutive odd numbers is x.What is the smallest of the three numbers?If we know the answer to the above question is 23, what is the value of unknown variable x? (we sample 10 backward chains and all predict x = 75, thus, no correct backward chains) Backward probability:Candidate answers generated by Self-Consistency: 21 (16 times), 23 (24 times) Forward probability: Combined probability