Enhance Large Language Models for Scientific Paper Question Answering
Zhonghan Chen, Yongquan Jiang, Yan Tao Yang, Xuanpei Jiang, Qiangwei Zhang · 2024
Recently, large language models (LLMs) have demonstrated promising applications across various domains, offering significant assistance to humanity. This study explores how to refine LLMs through fine-tuning to enhance their performance on scientific paper question answering tasks. We begin by collecting and constructing a composite task dataset that includes both textual and tabular data. To enhance the LLM's capabilities on composite tasks, we propose Mixture LoRA (MixLoRA) approach, which combines the advantages of composite task learning and parameter-efficient fine-tuning. Furthermore, we include general domain experts in MixLoRA to alleviate the issue of knowledge forgetting in the model in order to maintain its general capabilities. Comprehensive experiments are designed to validate the effectiveness of the method, with the results indicating that the approach enables the model to successfully learn downstream tasks with different data distributions while also maintaining the model's generalization abilities.