Comparative Evaluation of Vector Embedding Frameworks for Scalable Sematic Retrieval in PDF-Based RAG Systems
Olasehinde Omolayoa, Odunayo Babatope · International Journal of Research Publication and Reviews · 2025
We present a comparative analysis of using two different vector embedding frameworks for the implementation of Retrieval-Augmented Generation (RAG) pipelines to enable interactive question and answering result delivery over PDF documents containing structured text and tables.Both solutions adopt the LangChain framework with OpenAI's GPT-4o-mini for summarization and response generation, while leveraging Redis for storing raw documents for multi-turn conversation.We utilize two vector embedding storage strategy: the first pipeline utilises Facebook AI Similarity Search (FAISS), an in-memory vector similarity search engine optimized for local inference and the second employs the PostgreSQL extension, PGVector, for persistent, scalable vector storage.We implement a uniform document ingestion process, multi-vector retrieval mechanism, and a Streamlit-based chat interface across both systems.To evaluate their effectiveness, we propose an assessment framework measuring retrieval quality known as the generation quality (correctness, relevance, and faithfulness) using an LLM-as-a-Judge approach.The experimental results demonstrate that FAISS consistently delivers higher semantic retrieval precision and generation quality, with lower latency compared to PGVector.These findings suggest that FAISS is a more suitable choice for high-performance RAG pipelines, particularly in resourceconstrained environments.