Hashing-Based Multi-Modal Semantic Communication
Yibo Zhu, Hongyu Gu, Jiangtian Nie, Jianhang Tang, Jiangming Jin, Yang Zhang · 2024
The advanced sixth-generation (6G) wireless network is considered as an indispensable part of the Metaverse, where a substantial volume of communication content is transmitted through multiple modalities, placing significant transmission loads on communication channels. In this paper, we propose a framework for multi-modal semantic communication using hashing-based semantic extraction approach to produce optimal binary signatures (hash codes). Instead of directly using coarse-grained feature fusion methods, we capture deep semantics in self-attention manner, achieving fine-grained multi-modal feature fusion thereby strengthening the representation ability of hash codes. To enhance adaptability in practical situations, we then design a modality-completion module to address missing modalities in data, accommodating scenarios with both single-modal and cross-modal data. We evaluate the proposed semantic extraction framework on two popular multi-modal datasets, comparing it with the latest hashing methods and then demonstrate the effectiveness in various channel conditions.