Perceptual Visual Similarity from EEG: Prediction and Image Generation
Carlos de la Torre‐Ortiz, Tuukka Ruotsalo · 2024
Visual similarity estimation plays a fundamental role both in human cognition and multimedia information processing as it is the basis for many applications ranging from image search and recommendation to visual content generation. Existing computational models to assess visual similarity often diverge from human perception as they are typically trained solely on image data without information about how humans perceive image similarity. Here, we present an approach for learning perceptual visual similarity from brain recordings obtained via electroencephalogram (EEG). Our approach establishes a mapping between similarity reflected in the human cognitive system and a latent image representation. We evaluate the approach in two tasks. First, predicting visual distance from EEG data and second, adjusting a latent representation of a generative model to generate new images at a predicted distance from a given source image. Experiments demonstrate that the predicted distances from EEG closely align with the ground truth distances, and images generated using these predicted distances closely resemble the ground truth images. These findings open new possibilities for leveraging signals measured from human cognition to infer similarity as opposed to using only content-based models.