Finding unity in impact events: Amplitude envelope and multi-modal integration
Michael Schutz · 2015
Although we experience unitary perceptions, information from single events arrives via different physiological mechanisms. For example, a speech utterance is experienced concurrently by the visual (seeing the speaker’s lip movements) and auditory (hearing the utterance) systems. Conflicts in audio-visual information are generally resolved according to task type, with the visual system dominating spatial tasks and the auditory system temporal tasks. The theory of optimal integration (Ernst & Banks, 2002) refines this common observation by clarifying that task-specific integration reflects differences in modality acuity. Situations degrading the quality of one modality lead to a weakening of its traditional dominance (Alais & Burr, 2004) However, an illusion in which percussionists use physical gestures to control a listeners’ perception of note duration (Schutz & Lipscomb, 2007) breaks with this coherent framework. This novel documentation of a visual influence on a temporal judgments indicates that causal links between modalities alters integration processes (Schutz & Kubovy, 2009). Here we explore the auditory component of this illusion, demonstrating that pure tones shaped with exponentially decaying envelopes integrate with visual information mimicking an impact event, whereas spectrally matched pure tones shaped with the artificial amplitude envelopes frequently used in auditory research do not. Inconsistent with optimal integration, ratings of the tones in isolation are no more ambiguous for the visually-integration decaying tones than the sharply onset/offset “flat” tones. This presents a clear conflict with the widely accepted theory of “optimal integration” in which dominance is dictated by uni-modal information quality (Alais & Burr, 2004; Ernst & Banks, 2002). We have developed new software tools for this research affording simulation of biological motion as well as generation of spectrally matched tones with different amplitude envelope structures, which we have posted online at www.maplelab.net/software to aid other researchers.