FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang · 2024
Referring Expression Comprehension (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and languageto-image grounding.Consequently, it serves as an ideal testing ground for Multi-modal Large Language Models (MLLMs).In pursuit of this goal, we have established a new REC dataset characterized by two key features: Firstly, it is designed with controllable varying levels of difficulty, necessitating multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships.Secondly, it includes negative text and images created through fine-grained editing and generation based on existing data, thereby testing the model's ability to correctly reject scenarios where the target object is not visible in the image-an essential aspect often overlooked in existing datasets and approaches.Utilizing this high-quality dataset, we conducted comprehensive evaluations of both state-ofthe-art specialist models and MLLMs.Our findings indicate that there remains a significant gap in achieving satisfactory grounding performance.We anticipate that our dataset will inspire new approaches to enhance visual reasoning and develop more advanced cross-modal interaction strategies, ultimately unlocking the full potential of MLLMs.Our code and the datasets are available at https: //github.com/liujunzhuo/FineCops-Ref.