Performance Analysis of Large Language Models on Turkish Question-Answer Texts
Metin Bi̇lgi̇n, Mehmet BOZDEMİR, Engin DEMİR · 2024
One of the most studied topics in the field of natural language processing is question-answering systems. In recent years, finding the answer to a question asked within a given text has become increasingly important. In this study, a fine-tuning process was conducted on large language models using the SQuAD 1.1 dataset containing Turkish historical data. Five different large language models were fine-tuned in this study: "bert-base-uncased", "bert-base-turkish-cased", "distilbert- base-multilingual-cased", "mt5-base", and "mBart-large-50". While BERT and DistilBERT models use only the encoder part of the Transformer architecture, models like mT5 and mBART use both the encoder and decoder. It was ensured that most of the selected models were multilingual. Additionally, the performance of the non-multilingual "bert-base-uncased " model on Turkish data was also examined. To evaluate the study results, EM, F1, and Rouge metrics were used. As a result of the study, the "mBART" model achieved better results compared to the other models.