360-GAN: Cycle-Consistent GAN for Extrapolating 360-Degree Field-of-View
Jit Chatterjee, Maria Torres Vega · 2023
360-degree images, also known as panoramic, have become increasingly popular in the field of Extended Reality (XR). They offer an immersive experience to users, allowing them to explore images in a more engaging and dynamic manner. However, the visual quality associated with 360 images in XR can vary greatly depending on factors, such as image resolution with crisp details and vibrant colors. Thus, complex camera systems are required to shoot 360-degree environments. Generative adversarial networks (GANs), which have already been successfully applied to out-painting tasks and for the generation of masked regions in images, have the potential to solve the need for complex infrastructures. As such, from only a small RGB crop, the full environment could be generated. However, traditional GANs can fail to blend the input crop with the generated extrapolated region by introducing sharp vertical edges that disrupt the overall visual coherence. Another challenge in generating 360-degree images is the representation and handling of the spherical geometry of the panorama. In this work, we present 360-GAN, a cycle-consistent GAN model to generate 360-degree omnidirectional images from small RGB crops. Moreover, to maintain the spherical consistency of the generated 360 panoramic images, our method uses Structural Similarity Index (SSIM) as an added loss function. We evaluate our approach through quantitative measurements, benchmarking them against other state-of-the-art approaches. Our method generates realistic results maintaining the spherical consistency of the omnidirectional images with a Frechet Inception Distance (FID) of 46.59, nearly 6 points better than the most current state-of-the-art methods.