Paragraph vs Sentence in Automatic Question Generation Fine-Tuning using Text-to-Text Transfer Transformer for Bahasa Indonesia
Halim Wildan Awalurahman, Indra Darmawan Budi · 2024
Automatic Question Generation (AQG) has been developed to help create usable questions for assessment purposes. AQG has been adopted in many domains and languages, including Indonesia. The state-of-the-art method in AQG is the transformer model. The recent Indonesian AQG model, however, still inherits limitations in the form of irrelevant question-answer (QA) pairs. We propose different preprocessing mechanisms to reduce the irrelevant QA pairs utilising only sentence and top 3 sentences as the input, which has never been explored before. We used basic string matching and BM2SOkapi for this purpose. The multilingual Text-to-Text Transfer Transformer (mT5) base variant is fine-tuned in Indonesian SQuAD and TydiQA dataset with three different input schemes: paragraph, sentence, and top 3 most relevant sentences. We evaluated the model using BLEU and ROUGE metrics. Our findings suggest that different input scenarios can influence the performance of the model. The characteristic of the dataset also plays an important role in deciding which input scheme to use. Our findings could be the basis of further development for AQG in Indonesian, especially enhancing the preprocessing of the current and future models.