"Reasoning before Responding": Towards Legal Long-form Question Answering with Interpretability
Utkarsh Ujwal, Sai Sri Harsha Surampudi, Sayantan Mitra, Tulika Saha · 2024
Long-Form Question Answering (LFQA) represents a growing interest in Legal Natural Language Processing (Legal-NLP) as many individuals encounter legal disputes at some point in their lives, but lack of knowledge about how to negotiate these complex situations might put them at risk. The endeavor to generate detailed answers to contextually rich legal questions has faced challenges, primarily due to the limited availability of specialized datasets involving intensive manual effort or incapability of existing LFQA models to produce informative responses. Addressing this, our research introduces a semi-synthetic dataset, Legal-LFQA (L2FQA) created by exploiting a large language model (LLM) and utilizing contexts derived from existing legal datasets. Additionally, we hypothesize that integrating legal reasoning into the answer generation process of the LLMs will help bolster both the quality and interpretability of the produced responses. We systematically analyze the quality of L2FQA using human evaluation and natural language inference based metrics. Next, we benchmark L2FQA on a wide range of general-purpose and domain-specific LLMs using fine-tuning and in-context learning (with zero, one and few shot) strategies. The efficacy of these techniques is gauged through several automated and human evaluations. Results indicate that incorporating legal reasoning into the answer generation process provides an avenue for improving the quality of responses in the context of Legal-LFQA task. By addressing the challenges faced in LFQA and emphasizing the potential of interpretability, this research contributes to the foundational work in enhancing question-answering systems within the legal domain.