Fine Tuning LLMs for Low Resource Languages
Shreyas Joshi, Muhammad Shahnawaz Khan, Aditya Dafe, Kavita Singh, Vedant Zope, Tanish Jhamtani · 2024
Large Language Models (LLMs) hold immense potential, but their data hunger can limit its performance in processing languages with limited resources. This research study explores the techniques for fine-tuning LLMs specifically for low-resource settings. This study highlights the need for language-specific approaches that satisfy the unique characteristics of each language. Further, this study analyzes various fine-tuning techniques, including parameter-efficient methods (like Low Rank Adaption (LoRA) and Quantized Low Rank Adaption (QLoRA)), instruction tuning, and Representation Fine-Tuning (ReFT). By promoting these strategies, this work aims to bridge the language gap and empower low-resource languages within the LLM landscape.