Experimental semiotics applied to visual, sound, and musical structures
Roger A. Kendall, Scott D. Lipscomb · 2013
Abstract The relationship of music to film and multimedia has only recently received the attention of experimental psychologists and cognitive musicologists. This paper outlines theory, semiotical analysis, and experimental results using relations among variables of temporally organized visuals and music. First, a comparison and contrast is developed among the ideas in semiotics and experimental research, including both historical and recent developments. Then follows a musicological exploration. The resulting multidimensional structures of associative meanings, iconic meanings, and embodied meanings are applied to the analysis and interpretation of several classic film examples. Finally, experimental verification is provided. A series of experiments testing the perceptual fit of musical and visual patterns layered together in animations provided a determination of perceived goodness of fit between all pattern combinations, results of which confirmed aspects of the theory. However, exceptions were found when the complexity of the stratified stimuli possibly resulted in cognitive overload.