Finding the correspondence of audiovisual events by object manipulation
Kento Nishibori, Yoshinori Takeuchi, Tetsuya Matsumoto, Hiroaki Kudo, Noboru Ohnishi · Electronics and Communications in Japan · 2009
Abstract A human being understands the objects in the environment by integrating information obtained by the senses of sight, hearing, and touch. In this integration, active manipulation of objects plays an important role. We propose a method for finding the correspondence of audiovisual events by manipulating an object. The method uses the general grouping rules in Gestalt psychology, that is, “simultaneity” and “similarity” among motion command, sound onsets, and motion of the object in images. In experiments, we used a microphone, a camera, and a robot which has a hand manipulator. The robot grasps an object like a bell and shakes it or grasps an object like a stick and beats a drum in a periodic, or nonperiodic motion. Then the object emits periodic/nonperiodic events. To create a more realistic scenario, we put another event source (a metronome) in the environment. As a result, we had a success rate of 73.8% in finding the correspondence between audiovisual events (afferent signal) which are related to robot motion (efferent signal). © 2009 Wiley Periodicals, Inc. Electron Comm Jpn, 92(5): 1–13, 2009; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/ecj.10087