A Comparative Study of Prompting Strategies for Legal Text Classification
Ali Hakimi Parizi, Yuyang Liu, Prudhvi Nokku, Sina Gholamian, D. B. Emerson · 2023
In this study, we explore the performance of large language models (LLMs) using different prompt engineering approaches in the context of legal text classification.Prior research has demonstrated that various prompting techniques can improve the performance of a diverse array of tasks done by LLMs.However, in this research, we observe that professional documents, and in particular legal documents, pose unique challenges for LLMs.We experiment with several LLMs and various prompting techniques, including zero/few-shot prompting, prompt ensembling, chain-of-thought, and activation fine-tuning and compare the performance on legal datasets.Although the new generation of LLMs and prompt optimization techniques have been shown to improve generation and understanding of generic tasks, our findings suggest that such improvements may not readily transfer to other domains.Specifically, experiments indicate that not all prompting approaches and models are well-suited for the legal domain which involves complexities such as long documents and domain-specific language.