Sound localization and processing for inducing synesthetic experiences in Virtual Reality
Aleksei Tepljakov, Sergei Astapov, Eduard Petlenkov, Kristina Vassiljeva, Dirk Draheim · 2016
Synesthesia is the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Recent technological advances in the Virtual and Augmented Reality field allow to induce such experiences due to the effect of presence achieved in the virtual environment. In this paper, we present initial results related to a particular synesthetic experience; namely, we focus on the visual interpretation of sound. Towards that end, we apply a series of recently developed methods for detecting the source of sound in a three-dimensional space around the listener, and transform the audio signal into a series of visual shapes, such that the size of each shape is determined by the loudness of the sound source, and its color is determined by the dominant spectral component of the sound. Experimental results are provided and discussed.