A framework for evaluation and requirement extraction for fine-tuning of Large Language Models in multimodal medical diagnosis
Dimitrios P. Panagoulias, Anastasios F. Palamidas, Maria K. Virvou, George A. Tsihrintzis · Knowledge-Based Systems · 2025
Objective: Large language models constitute a breakthrough state-of-the-art Artificial Intelligence technology which is rapidly evolving and promises to aid in medical diagnosis. In this study, we propose a novel evaluation framework for extraction of fine-tuning requirements based on the Objective Structured Clinical Examinations (OSCE) that can increase LLM potential and applicability. Methods: We developed an OSCE based evaluation meta-framework leveraging IoT-based data retrieval with a two-step approach designed to analyze and guide improvement of LLMs in multimodal medical diagnosis: (1) structured interaction evaluation and (2) domain-specific analysis of extracted data. Using Image-Metadata Analysis (IMA), Named Entity Recognition (NER), and Knowledge Graphs (KG), this framework identifies image domains, extracts relevant entities, and assesses connections in KGs. These methods collectively reveal areas for improvement, guiding fine-tuning to enhance diagnostic accuracy and contextual understanding in medical applications. Results: Using this paradigm, (1) we evaluate the correctness and accuracy of generated medical diagnosis with publicly available multimodal-multiple-choice-questions in the vast domain of General Pathology and (2) proceed to the domain-specific analysis. We identify and visualize the model performance across specific organs, diseases, and pathological themes, detecting areas of lower accuracy, such as in cardiovascular conditions like atherosclerosis. This targeted approach enables precision-focused fine-tuning, applying additional data to specific weaknesses rather than a broad, generalized tuning across all pathology. Contributions: Our framework’s primary contribution is its OSCE-inspired ability to dynamically identify and target under-performing areas within a broad domain, enhancing fine-tuning efficiency and diagnostic accuracy in a resource-effective and iterative manner removing the dependency on bulk adjustments, making it particularly suitable for sensitive applications where precision and resource efficiency are essential, such as in medical diagnostics.