Evaluating Consistency of Image Generation Models with Vector Similarity
C. B. Pronin, Aleksandr A. Podberezkin, Борзенков Александр Михайлович · 2024
This paper presented an approach for evaluating image generation models by measuring their consistency in producing similar results across multiple prompts and seeds. We proposed a method that utilizes image vectorization models to convert generated images into vectors, which are then compared using similarity scores. By analyzing the consistency of generation results, we aimed to provide insights into the randomness of image generation and its potential impact on tasks such as animation, concept art, and blueprint creation. Our findings offer valuable considerations for practitioners working with image generation models, particularly when consistency is crucial for specific applications. We calculated the impact of seed randomness and prompt variation on generation consistency, providing insights into the stability and reliability of image generation. Our results show that with a random seed using a highly descriptive prompt approach generally leads to more consistent generation results compared to the incremented seed, where the difference in consistency was mostly negligible. The proposed evaluation technique leverages advanced image-to-vector conversion techniques to transform generated images into numerical representations. These vectorized outputs are subsequently analyzed using sophisticated similarity metrics, allowing researchers to quantify the degree of consistency between images produced under different conditions. By systematically comparing the output of image generation models under multiple prompts and random seed values, this investigation aims to shed light on the inherent randomness present in current state-of-the-art image synthesis methods. The findings of this study have significant implications for various applications where consistency is paramount, including high-fidelity animation production, conceptual artwork creation, and architectural blueprint design. However, our proposed method also has limitations to consider. The choice of image vectorization model and similarity metric can affect the results, so it is important to make all comparable tests with the same img2vec model. Future research in this field could be used to better understand how consistency in image generation relates to other factors such as model architecture, training data, and prompt complexity.