SSR : SAM is a Strong Regularizer for domain adaptive semantic segmentation
Yanqi Ge, Ye Huang, Wen Li, Lixin Duan · 2024
We introduced SSR, which utilizes SAM (segment-anything) as a strong regularizer during training, to greatly enhance the robustness of the image encoder for handling various domains. Specifically, given the fact that SAM is pre-trained with a massive-scale dataset that covers a diverse variety of domains, the feature encoding extracted by the SAM is obviously less dependent on specific domains when compared to the traditional ImageNet pre-trained image encoder. Meanwhile, the ImageNet pre-trained image encoder is still a mature choice of backbone for the semantic segmentation task, especially when the SAM is category-irrelevant. As a result, our SSR provides a simple yet highly effective design. It uses the ImageNet pre-trained image encoder as the backbone, and the intermediate feature of each stage (i.e. there are 4 stages in MiT-B5) is regularized by SAM during training. Extensive experiments show our SSR significantly improved performance over the baseline without introducing any extra inference overhead.