SeerahGPT: Retrieval Augmented Generation based Large Language Model
Surayya Obaid, Narmeen Zakaria Bawany · 2024
Large Language Models(LLMs) have become widely recognized in recent years for their exceptional performance in language generation capabilities. As a result, an unprecedented rise is seen in its use cases in various domains specifically involving Natural Language Processing(NLP). These models however perform suboptimally when exploited in the field of studies where authenticity of the generated content is a critical aspect. One such domain is usage of LLM for exploration of the life of Prophet Muhammad S.A.W(commonly referred to as Seerah). It is of utmost significance to ensure the authenticity and reliability in the sources used and reported by the LLM due to the sensitive nature of the domain. The contemporary LLMs, however, lack the explainability in their response due to their inherent black-box nature. In our study, we have presented a novel LLM named SeerahGPT that addresses this challenge with the help of retrieval-augmented generation (RAG). This technique enables the model to utilize both parametric and nonparametric memories for generating response of queries. Our model, built on the Llama-2-7b architecture, employs Sentence Transformer embedding to effectively retrieve relevant information. The model's capabilities are augmented by integrating it with a corpus having Islamic texts such as the Quranic translation and Hadith collections, and historical accounts. The model's performance is benchmarked against its base model using both quantitative and qualitative metrics. The comparative analysis with Llama-2-7b revealed that SeerahGPT incorporation with external knowledge sources, provided more authentic and verifiable responses, despite the others exhibiting greater fluency. Performance metrics such as BLEU, ROUGE, and METEOR indicated SeerahGPT's better accuracy and contextual handling. This study paves way for analysis of such sensitive domains in more efficient way that can be utilized in other complex domains such as Islamic theology and Fiqh or legal system.