Evaluation metrics for galaxy image generators
Stefan Hackstein, Vitaliy Kinakh, C. Bailer, M. Melchior · Astronomy and Computing · 2023
A major problem with deep generative models is verifying that the generated distribution resembles the target distribution while the individual generated sample is indistinguishable from the original data. In particular, for application in astrophysics we need to be sure that the generated data matches our prior knowledge and that the generated samples entail all object types with the correct frequency and diversity. We currently lack objective ways to systematically assess these quality aspects, where human inspection reaches its limits, as this requires detailed analysis of a large data volume. In this work, we identify reasonable metrics for the quality of galaxy image generators. To this end, we compare a small set of conditional image generators, trained on galaxy images with classification labels for visual morphology features. Our main contribution is a new set of cluster-based metrics for matching the generated distribution to the target distribution. Furthermore, we use the Wasserstein distance on proxies for galaxy morphology as well as a number of other metrics commonly used for image generators. The newly introduced cluster-based metrics are good proxies for the quality of the generated distribution and are suited for automatized identification of mode collapse. Furthermore, the cluster metrics allow for a qualitative interpretation of the generated distribution. The metrics based on morphological statistics provide a useful tool to probe the physical soundness of generated samples. Finally, we find that kernel inception distance used with an InceptionV3 model pre-trained on ImageNet is a good proxy for the overall quality of galaxy image generators, although it cannot be interpreted that easily.