Fast transform type selection using conditional Laplacian distribution based rate estimation
Bohan Li, Jingning Han, Yaowu Xu · 2020
Selecting among multiple transform kernels to code prediction residuals is widely used for better compression efficiency. Conventionally, the encoder performs trials of each transform to estimate the rate-distortion (R-D) cost. However, such an exhaustive approach suffers from a significant increase of complexity. In this paper, a novel rate estimation approach is proposed to by-pass the entropy coding process for each transform type using the conditional Laplace distribution model. The proposed method estimates the Laplace distribution parameter by the context inferred by the quantization level and finds the expected rate of the coefficients for transform type selection. Furthermore, a greedy search algorithm for separable transforms is also presented to further accelerate the process. Experimental results show that the transform type selection scheme using the proposed rate estimation method achieves high accuracy and provides a satisfactory speed-performance trade-off.