Face Restoration-Based Scalable Quality Coding for Video Conferencing
Wei Jiang, Hyomin Choi, Fabien Racapé, Simon Feltman, Fatih Kamışlı · 2023
We investigate a new scalable framework for human face coding, where the reconstruction at the decoder synthesizes the output of two coding layers. The base layer embeds the input into a discrete code space described by a learned codebook and transmits the codebook indices to the decoder, from which the output of the base layer is generated. In addition, the input is optionally rescaled and coded using a neural network (NN)-based coder for the enhancement layer to add supplemental information from the original input to the generative output from the base layer. The final reconstruction then synthesizes the outputs from the two layers using a learned NN module with a weighting parameter. Furthermore, we apply an online learning scheme to the enhancement layer along with the weighting parameter to adaptively optimize the rate and distortion based on input. Experimental results show superior coding gains in various metrics as well as noticeable visual improvements with our method at the extremely lower bits per pixel compared to the latest H.266/VVC standard.