Cura-LLaMA: Evaluating open-source large language Model’s question answering capability on medical domain
Jiean Zhu · Applied and Computational Engineering · 2024
This paper presents the development and evaluation of “Cura-LLaMA,” an open-source large language model (LLM) tailored for the medical domain. Based on the TinyLlama model, Cura-LLaMA was fine-tuned using the PubMedQA dataset to enhance its ability to address complex medical queries. The model’s performance was compared with the original TinyLlama, focusing on its accuracy and relevance in medical question-answering tasks. Despite improvements, the study highlights challenges in using keyword detection methods for evaluation and the limitations of omitting non-essential columns during fine-tuning. The findings underscore the potential of fine-tuning open-source models for specialized applications, particularly in resource-limited settings, while pointing to the need for more sophisticated evaluation metrics and comprehensive datasets to further enhance accuracy and