Task Knowledge Injection: Training-Free Adaptation of Multimodal Large Language Models for Remote Sensing Image Understanding
Haifeng Li, Qiujun Li, Cheng Hong Yang, Wang Guo, Mengyao Li, Hongyuan Yuan, Run Shao, Chengli Peng · IEEE Geoscience and Remote Sensing Letters · 2025
Parameter fine-tuning is the mainstream approach for adapting Multimodal Large Langauge Models (MLLMs) to downstream remote sensing tasks. However, such method risks degrading pre-trained knowledge and also incur significant costs. This paper argues that downstream adaptation of MLLMs essentially involves effective injection of task-specific knowledge, which does not necessarily require parameter updates. Based on this perspective, we propose a training-free knowledge injection method. By constructing a multi-task knowledge base (MTKB), the model can dynamically retrieve task-related knowledge to serve as context during inference, thereby enhancing its understanding. Specifically, we design a three-part framework. (1) Task knowledge construction: Diverse texts are unified into a key and value structure for image-text matching, forming the MTKB. (2) Two-stage retrieval: A coarse-to-fine process is employed to match query images with task knowledge in the MTKB, utilizing both unimodal and cross-modal similarity. (3) Knowledge injection: Matched knowledge is integrated into the MLLM via extended embeddings, without altering parameters. Experimental results across multiple datasets demonstrate that our method significantly enhances the image understanding capabilities of the model. Our approach achieves about 5% improvement in accuracy and related metrics across several datasets, with performance on the RSVQ-LR dataset comparable to specialized models.