LLaVA-Mammo: adapting LLaVA for interactive and interpretable breast cancer assessment
Xuxin Chen, Jingchu Chen, Xiaoqian Chen, Judy Wawira Gichoya, Hari Trivedi, Xiaofeng Yang · 2025
Breast cancer remains a leading health concern for women globally, with mammography serving as the primary screening tool. While deep learning (DL) based computer-aided detection/diagnosis (CAD) systems have shown promise, they function as "black boxes" with limited interaction, hindering clinical adoption. This study introduces LLaVA-Mammo, an adaptation of the Large Language and Vision Assistant (LLaVA) for breast cancer assessment that addresses these limitations through two key features: interpretability by communicating findings in natural language that describes image features radiologists recognize, and interactivity through dialogue that allows users to pose follow-up questions. Testing LLaVA-Mammo on BI-RADS breast density categorization and malignancy classification tasks, we found that increasing language model size from 7 billion (B) to 13B parameters improved performance, with density categorization accuracy rising from 72.1% to 76.6% and malignancy classification AUC increasing from 0.687 to 0.723. These results indicate that LLaVA-Mammo marks a significant shift in AI-assisted mammography by providing clear, interactive outputs that foster collaboration between radiologists and AI in clinically aligned decision-making.