Enhancing LLM Performance on Legal Textual Entailment with Few-Shot CoT-based RAG
Sneha Ann Reji, Reshma Sheik, Amiel T. Sharon, Avisha Rai M, S. Jaya Nirmala · 2024
Large Language Models (LLMs), leveraging extensive data, have become successful in text generation applications. However, their integration into the legal domain presents distinct challenges, prompting deviations in thought processes and potentially misleading outcomes. Such discrepancies can significantly impact the efficacy of legal text applications. In our study, we propose a novel approach that combines retrieval-based methods with generation-based techniques to provide relevant explanations and labels for the given query for legal textual entailment task, leveraging the concept of Chain of Thought (CoT) for reasoning and few-shot prompting. Later, we enhance LLM outputs through the application of various text generation strategies. Our findings demonstrate an enhancement, with a 5% increase in F1 score and a 3% improvement in accuracy when compared with zero-shot baselines. This highlights the effectiveness of utilizing these strategies in specialized fields such as law, where precision and reliability are of utmost importance.