Fine-Tuning for Question Answering in Low-Resource Languages: A Case Study on Khmer
Kimleang Ly, Dona Valy, Phutphalla Kong · 2024
With the rise of artificial intelligence, a large language model can understand and generate human language by training it with massive amounts of text data. They can be fine-tuned to a specific task like question answering, summarization, etc. However, most of the existing pre-trained models nowadays were trained with English datasets, leading to limited support and low performance in other languages especially low-resource languages like Khmer. To address the imbalance, we aim to build a Khmer language model by investigating the effectiveness of fine-tuning large language models. The full fine-tuning applies to Qwen2 (0.5B, 1.5B) and Gemma2 2B. Fine-tuning with low-range adaptation (LoRA) applies to Mistral 7B, Gemma 7B, and Qwen2 7B. We collect datasets from online sources containing question-answer pairs in the general knowledge domain. The vocabulary expansion employs to model that lack of Khmer token representation. To achieve our experiment. The recall-oriented understudy for gisting evaluation (ROUGE) and bilingual eval-uation understudy (BLEU) as the evaluation metrics to measure the similarity between generated and reference responses. As a result, the Qwen2 1.5B archives the highest score in both metrics among small-scale model sizes and its original model, while the Mistral 7B model outperforms other 7B models. This research demonstrates that the fine-tuning strategy enhanced model performance across different model sizes, even when trained on a limited dataset of a downstream task.