Prompt-Based Memory Bank for Continual Test-Time Domain Adaptation in Vision-Language Models
Ran Wang, Hua Zuo, Zhen Fang, Jie Lü · 2024
In dynamic environments, the generalization capabilities of large-scale vision language models tend to decline. This is attributed to the evolving distribution of target domains over time, leading to misalignment between image and text pairings, affecting the model’s performance. Addressing this, Test-Time Adaptation (TTA) has been proposed to adapt pre-trained source models to these changing target domains during testing phases. However, traditional TTA approaches, which are designed for a single changing scenario and mainly depend on self-training and entropy minimization, are easily affected by extreme and novel samples in long-term environments, leading to error accumulation and catastrophic forgetting. Although previous Continual Test-Time Adaptation (Continual TTA) methods based on the teacher-student framework can effectively address long-term adaptation issues, they are not feasible for large-scale vision language models due to their high memory requirements. To overcome these challenges, we introduce a novel approach: Prompt-based memory bank for Continual Test-Time Adaptation (PCoTTA). PCoTTA uniquely freezes the CLIP image and text encoders, focusing on updating and storing trainable prompts, significantly reducing memory usage. By implementing a stable pseudo-label strategy and high gradient sensitivity updating, PCoTTA effectively learns new knowledge. In long-term dynamically changing environments, PCoTTA demonstrates high stability and accuracy and achieves a good balance between learning new information and retaining existing knowledge, significantly enhancing the adaptability and generalization capabilities of the CLIP model. Through extensive experimental comparisons, PCoTTA surpasses the current state-of-the-art methods, achieving an average 2% improvement in accuracy for both test-time adaptation and continual test-time adaptation tasks.