Open-RAG: Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models
Shayekh Bin Islam, M. A. Rahman, KM Hossain, Enamul Hoque, Shafiq Joty, Md Rizwan Parvez · 2024
Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs) , but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs.To mitigate this gap, we introduce a novel framework, OPEN-RAG, designed to enhance reasoning capabilities in RAG with opensource LLMs.Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single-and multi-hop queries.OPEN-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading.As a result, OPEN-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses.In addition, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed.Experimental results show that the Llama2-7Bbased OPEN-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT, Self-RAG, and Command R+ in various knowledgeintensive tasks.We open-source our code and models at https://openragmoe.github.io/Q: According to the 2010 census, what was the population of the city in which Andover USD 385 is located?Adaptive Retrieval Knowledge 1: Andover, Kansas Andover is a city in Butler County, Kansas, United States, and a suburb of Wichita.As of the 2010 census, the city population was 11,791.