Text Augmented Spatial Aware Zero-shot Referring Image Segmentation
Yucheng Suo, Linchao Zhu, Yi Ping Yang · 2023
In this paper, we study a challenging task of zero-shot referring image segmentation.This task aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations.Previous research takes advantage of pre-trained crossmodal models, e.g., CLIP, to align instancelevel masks with referring expressions.Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions.To address this challenge, we introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework that is training-free and robust to various visual encoders.TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial rectifier for mask post-processing.Notably, the text-augmented visual-text matching score leverages a P -score and an N -score in addition to the typical visual-text matching score.The P -score is utilized to close the visual-text domain gap through a surrogate captioning model, where the score is computed between the surrogate model-generated texts and the referring expression.The N -score considers the fine-grained alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from the mined distracting phrases.Extensive experiments are conducted on various datasets, including Re-fCOCO, RefCOCO+, and RefCOCOg.The proposed method clearly outperforms state-ofthe-art zero-shot referring image segmentation methods.