ACTOR: Adapting CLIP for Fully Transformer-based Open-vocabulary Detection
Zhaopeng Yang, Huan Hao, Shangsong Liang · 2024
Open-vocabulary detection (OVD) aims to identify objects from novel, unseen categories that extend beyond the base categories encountered during training. Recent approaches generally resort to large-scale Visual-Language Models (VLMs), such as CLIP, to identify novel objects. However, incorporating these models into the OVD problem faces two main challenges: (1) the dgenerated performance when applying a VLM that pre-trained for whole image classification on regional object recognition; (2) the difficulty to localize objects from novel categories. To overcome the above challenges, we propose ACTOR, a DETR-style model which Adapting CLIP for fully Transformer-based Open-vocabulary detection via bidiRectional prompt learning and conditional decoding. Bidirectional prompt learning tightens the alignment between CLIP's regional object features and text embeddings. Conditional decoding facilitates learning a generalizable object localizer through a class-aware matching mechanism. Our experiments on the OV-COCO benchmark demonstrate that ACTOR achieves a competitive detection performance with the end-to-end open-vocabulary detector OV-DETR, while exhibiting a much faster inference speed.