Semantic-Topology Preserving Quantization of Word Embeddings for Human-to-Machine Communications
Zhenyi Lin, Lin Yang, Yi Gong, Kaibin Huang · IEEE Transactions on Communications · 2024
The vision of 6G mobile networks aims to connect intelligent machines to humans to provide the latter with cooperation, care, and assistance. The mainstream approach for human-to-machine (H2M) semantic communication is to map words into (word) embedding vectors which are clustered according to their semantic similarity to facilitate machines’ interpretation of human languages. The computation-intensive tasks of text-to-embedding mapping are usually delegated to an edge server that senses human commands, maps them into embedding vectors, and then transmits the vectors to a machine over a wireless link. In this work, we propose a quantization framework customized for embedding vectors, called semantic-topology preserving VQ (SemTop-VQ), to overcome the communication bottleneck due to the vectors’ high dimensionality. While traditional VQ focuses on minimizing the distortion of individual vectors, SemTop-VQ aims to minimize the distortion of the topology of embedding matrix, referring to the vectors’ relative positions that represent semantics. To this end, we adopt a topology-distortion metric, termed pointwise-inner-product (PIP) loss, a hierarchical VQ architecture targeting high-dimensional VQ. In this architecture, an embedding vector is decomposed into blocks; the norm and shape (normalized vector) are quantized separately using a scalar and a Grassmannian quantizers, respectively. The main feature of SemTop-VQ lies in deriving from the PIP loss a set of so-called semantic-importance indicators, which reflect the level of influences of individual blocks’ quantization errors on the topology distortion. Then the indicators are applied to optimize quantization-bit allocation for decomposed vector blocks under the criterion of PIP-loss minimization. In practice, the usage probabilities of embedding vectors for a specific machine task are highly skewed and the task is time-varying. We exploit this fact to further develop SemTop-VQ to feature task adaptation that can attain a higher communication efficiency. The task-adaptive VQ is realized via the use of a frequently used (quantization) codebook that is much smaller in size than the original codebook and continuously updated via estimation of embedding-usage distribution. Our experiments using real embedding datasets, namely Word2Vec and Glove, demonstrate the effectiveness of SemTop-VQ as a goal-oriented technique for efficient H2M communications.