Parallelized Nonlinear Scaled Transform for HEVC
Pierre-Alain Afro, Loïc Strus, Hugo Chauvet, Laurent Bonnaud, Alice Caplier, Frédéric Robin · 2024
Transform coding is widely used for compressing video data by compacting energy in the frequency domain. Conventional codecs like HEVC rely on the Discrete Cosine Transform, which has demonstrated a favorable balance between performance and complexity. The emergence of deep learning in video coding has prompted exploration into nonlinear transforms (NLT) to capitalize on nonlinearity for non-stationary signal sources. NLT, trained to minimize rate-distortion, surpass conventional linear transforms. However, these approaches require modifications to existing standards, making them incompatible with current codecs. Additionally, scalar quantization (SQ) can be substituted with an optimized scheme like rate-distortion optimized quantization (RDOQ). While RDOQ yields satisfactory results, its iterative nature poses challenges for hardware integration. Although DL-based RDOQ have been proposed to tackle this issue, they are not directly trained to minimize a rate-distortion function and the transform operation is not optimized. For that purpose, we propose a parallelized nonlinear scaled transform trained for replacing and improving forward DCT and SQ of HEVC while ensuring standard compliance. Our proposed method is tested on 4×4 and 8×8 blocks and reaches 1.13 % BD-rate reduction on luma, up to 1.85 %.