Knowledge Graph-Enhanced Artwork Image Captioning
Can Yang · 2024
This paper explores the unique challenges of artwork image captioning, a task that demands deep understanding of historical, cultural, and stylistic elements often absent in traditional image captioning. We conducted preliminary experiments using a Meshed Memory Transformer on the Iconclass AI Test Set, which revealed significant improvements in standard metrics but highlighted critical limitations in current datasets and evaluation methods. To address these issues, we propose a novel approach integrating knowledge graphs with large language models. This approach involves creating a specialized art ontology and knowledge graph, and developing new evaluation metrics specifically designed for artwork captioning. While not yet implemented, this proposed method aims to generate more comprehensive, contextually rich, and accurate captions for artwork images. Our research lays the groundwork for future advancements in artwork image captioning, potentially enhancing the accessibility and educational value of digital art collections.