Dual-Scale Transformer with Variable Bitrate Synchronization for Neural Video Compression
Yiming Wang, Yaojun Wu, Zhaobin Zhang, Qian Huang, Bin Tang, Zhangjing Yang, Kai Zhang, Li Zhang · ACM Transactions on Multimedia Computing Communications and Applications · 2026
Neural video compression (NVC) has emerged as a promising paradigm for improving rate-distortion performance. However, existing neural video codecs predominantly rely on convolutional neural networks (CNNs) with limited local receptive fields to generate the latent representations, often neglecting global–local spatial correlations. This leads to suboptimal feature modeling and redundancy in the latent space. To address this limitation, we propose a novel Dual-Scale Transformer (DST) block specifically tailored for NVC, which effectively enhances coding efficiency. The DST block incorporates a Global–Local (Shifted) Window-based Self-Attention (GL(S)WSA) mechanism to jointly capture global structure information and local texture details. Moreover, we design a Cross-Gated Feed-Forward Network (CGFFN) to adaptively modulate complementary components, producing more compact and expressive latent representations. Furthermore, to overcome the drawbacks of traditional asynchronous training and further boost rate-distortion performance, we introduce a Variable Bitrate Synchronization (VBRS) strategy that leverages multi-GPU parallel training, with each GPU dedicated to a specific bitrate and synchronized via gradient backpropagation for joint optimization. Experimental results demonstrate that our proposed method achieves the higher coding performance compared to the previous state-of-the-art (SOTA) methods and significantly outperforms H.266/VVC (VTM-13.2) under various low delay B (LDB) coding configurations.