Multimodal Knowledge Graph Embedding With Self-Attention Graph Neural Network
Wei Zhang, Tingting Liu, Ziming Li, Yan Dai, Xu Zhou · IEEE Transactions on Computational Social Systems · 2025
Multimodal knowledge graphs are typically represented as heterogeneous graphs, serving as crucial structures for depicting data that integrates various modalities, such as text, images, and audio, within multifaceted relationships. Established knowledge representation methods have aimed to enhance the expressive capabilities of knowledge triplets in these graphs by increasing the interaction between entity and relationship embeddings. However, these methods often fall short in encapsulating the multilayered semantics of multimodal data, especially considering the complex association attributes an entity may possess under specific relations. Graph neural networks (GNNs) provide a mechanism to weight neighboring nodes of entities based on structural and multimodal information. Yet, their approach lacks precision when dealing with the intricate interactions between an entity and its neighboring nodes, especially when these interactions involve diverse data types. Addressing this shortcoming, we introduce the self-attention graph neural network model (SAGNN). This model implements self-attention mechanisms for composite entities within multimodal KGs, allowing it to aggregate information from neighboring nodes effectively. By doing so, it dynamically updates entity representations based on the varying contribution levels of neighbors, considering both textual and nontextual data. The encoder of the SAGNN model is structured with multiple graph attention mechanism layers, designed to handle the multilocal features of composite entities in a multimodal context. Consequently, it adapts to learn composite entity embeddings that effectively integrate information from various data types. The decoder, on the other hand, is geared towards capturing the graph features during the decoding of triplets, considering the structure aspect of the knowledge graph. In tasks related to knowledge graph completion, particularly in multimodal settings, empirical results reveal significant improvements using the SAGNN model across all datasets.