BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering
Taolin Zhang, Dongyang Li, Qizhou Chen, Chengyu Wang, Xiaofeng He · 2025
Multi-hop question answering (QA) involves finding multiple relevant passages and performing step-by-step reasoning to answer complex questions.Previous works on multi-hop QA employ specific methods from different modeling perspectives based on large language models (LLMs), regardless of question types.In this paper, we first conduct an in-depth analysis of public multi-hop QA benchmarks, categorizing questions into four types and evaluating five types of cutting-edge methods: Chainof-Thought (CoT), Single-step, Iterative-step, Sub-step, and Adaptive-step.We find that different types of multi-hop questions exhibit varying degrees of sensitivity to different types of methods.Thus, we propose a Bi-levEL muLti-agEnt reasoning (BELLE) framework to address multi-hop QA by specifically focusing on the correspondence between question types and methods, with each type of method regarded as an "operator" by prompting LLMs differently.The first level of BELLE includes multiple agents that debate to formulate an executable plan of combined "operators" to address the multi-hop QA task comprehensively.During the debate, in addition to the basic roles of affirmative debater, negative debater, and judge, at the second level, we further leverage fast and slow debaters to monitor whether changes in viewpoints are reasonable.Extensive experiments demonstrate that BELLE significantly outperforms strong baselines in various datasets.Additionally, the model consumption of BELLE is higher cost-effectiveness than that of single models in more complex multihop QA scenarios.(B) Single-step [Multi-Hop Question:] What was the former band of the member of Mother Love Bone who died just before the release of Apple?[Answer:] Malfunkshun Multi-Hop Question Retrieval Docs Multi-Hop Answer (C) Iterative-step (D) Sub-step Multi-Hop Question (Intermediate) K times Multi-Hop Answer Multi-Hop Question 1 2 3 A1 A2 A3 Multi-Hop Answer (E) Adaptive-step Multi-Hop Question Classifier (D) Sub-Step (B) Single-step (A) CoT ✔ (A) CoT (C) Iterative-step Let's think step-by-step Multi-Hop Question Multi-Hop Question Retrieval-augmented Reasoning Closed-book Reasoning Our Agent-Based Reasoning Operators Pool …… Multi-Hop Question Inference Comparison Temporal Null 1. Use Sub-Step to decompose query 2. Use Single-Step to retrieve subquery 3. Aggregate the sub-answer Execution Plan Agents Multi-hop QA Task Environ -ment interaction invoke solve