Extended StyleGAN Encoder for Image Restoration
Kaitong Lin, Qingling Cai · 2022 26th International Conference on Pattern Recognition (ICPR) · 2022
Great success has been achieved in reusing Style-GAN, but there are still challenges in reusing StyleGAN to image restoration tasks. We believe that the limited expression of StyleGAN’s latent code hinders its effective use in image restoration tasks. Therefore, we propose to extend StyleGAN’s latent code to improve its expressiveness. Specifically, we propose a new model called Extended StyleGAN Encoder (ESE). ESE is based on Feature Pyramid Network, which can encode input images both in coarse-grain and fine-grain, and generate H × W × 2C feature maps instead of 2C-dimensional style vectors. In this way, ESE can provide more information of the input images to StyleGAN, which can achieve a more accurate reconstruction. Therefore, ESE can be applied to different image restoration tasks. Experiments show that ESE can surpass some models specifically designed for regular and irregular mask inpainting tasks, while it is difficult for other GAN Inversion methods. And experiments also show that ESE achieved good performance on colorization and denoising tasks. Besides, we further analyze the impact of the extended latent code by visualizing the outputs of ESE and the intermediate outputs of StyleGAN, which demonstrates that ESE does really reuse the knowledge of StyleGAN.