Enhancing Interactive Querying with a Multimodal RAG System: Integrating Text, Video, and Document Analysis via LLaMA3
Anjali Uday Patel, Rajpurohit Shivani, N. Usha, A Shruthiba · 2025
This paper introduces a sophisticated Retrieval-Augmented Generation (RAG) System designed to streamline the interaction and information retrieval across diverse data formats, including documents, videos, images, and audio, both readable and non-parsable. Utilizing the cutting-edge capabilities of the LLaMA3 language model, our system enables robust handling of multimedia queries through a finely tuned embedding process. The system is enhanced with OCR capabilities to transform non-parsable PDFs and handwritten documents into searchable content, coupled with a Milvus vector database for efficient similarity searches. Our architecture integrates MongoDB for structured data management and Redis for enhanced caching mechanisms, ensuring high performance and scalability. The system is equipped with API endpoints that support seamless user interactions through a web-based interface developed in HTML and J avaScript. Extensive testing and comprehensive API documentation are performed using Swagger to guarantee reliability and ease of use. Preliminary results demonstrate the system's efficacy in handling complex multimodal queries involving text, images, video, and audio with high precision, achieving significant advancements over traditional querying systems. The RAG system not only simplifies data interaction but also enhances the accessibility and manageability of information. Future advancements will focus on incorporating real-time contextual learning, expanding support for dynamic, complex queries across various domains, and optimizing deployment for edge devices, ensuring seamless multimodal interaction even in resource-constrained environments. Our implementation showcases the transformative potential of advanced language models in revolutionizing data interaction and retrieval in digital environments.