Robust Semantic-Aware Neural Video Compression
Hao Cao, Jiangkai Ying, Zhijin Qin · 2024
Reducing usage, lowering processing delay, and enhancing resistance to transmission errors are critical issues that must be addressed in practical system deployments of video codecs. However, neural video codecs (NVCs) suffer from high encoding complexity and are vulnerable to transmission errors. To address these issues, we propose a robust semantic-aware neural video codec (RSA-NVC). Specifically, at the encoder side we first extract the global semantic information from each frame, then divide the frame into segments and allocate different compression rate for each segment based on its contribution to global semantic information. A NVC backbone is followed to compress each segment. We also propose a learnable boundary-aware deblocking filter to alleviate blocking artifacts caused by the segmentation operation. At the decoder side, we propose an auxiliary-predictive (AP) frame-based reference updating scheme, which could combat transmission errors with low complexity by synchronizing references between the encoder and the decoder for inter-frame prediction. Simulation results indicate that the RSA-NVC could effectively lower the data traffic by 33.49% under the same accuracy for video instance segmentation, and reduce the overhead required for combating transmission errors by 60.53% compared to the state-of-the-art NVC.