InvaderDefender: Multimodal recognition of invasive alien species via a large language model with vision-guided targeted RAG
Wenda Luo, Xiaoyu Zhang, Zhibo Chen, Guangyu Huo, Rui Zhang, Liping Mu · Expert Systems with Applications · 2025
Invasive Alien Species (IAS) pose a significant threat to global biodiversity and economies, necessitating accurate and efficient identification for effective management. Traditional methods are often constrained by time, cost, and expertise while existing unimodal deep learning approaches struggle with species phenotypic diversity, environmental complexity, and long-tailed data distributions. This study introduces InvaderDefender, a novel “Visual Recognition-Semantic Enhancement” two-stage multimodal fusion framework to address these challenges. The first stage utilizes an EfficientNetV2-L visual backbone, enhanced with a novel wavelet fusion module for fine-grained feature extraction, and a combined strategy of class-aware data resampling and gradient-aware focal loss to address long-tailed distributions. The second stage inputs the Top-k probability outputs from the visual model, along with user-provided text, into the GLM-4 LLM enhanced by our proposed Vision-Guided Targeted Retrieval Augmented Generation (VGT-RAG). This GLM-4 leverages a specially constructed expert knowledge base for 20 target IAS to perform in-depth inference and output the final identification. To support this research, the first multimodal benchmark dataset specifically for these IAS was constructed. InvaderDefender achieves a Top-1 accuracy of 99.31%, significantly outperforming mainstream unimodal models. Compared to the visual-only baseline, the full multimodal framework boosts Top-1 and Mean Accuracy by 0.90 and 2.15 percentage points, respectively, ensuring a more balanced performance across all species, particularly rare ones. This work establishes an efficient and robust framework for IAS identification, providing a novel methodology for integrating multimodal learning and knowledge-driven AI in biodiversity monitoring.