Multimodal Looper: A Live-Looping System for Gestural and Audio-visual Improvisation
Pelin Kiliboz, Cumhur Erkut · 2024
We present the Multimodal Looper, an embodied digital interface that connects body movements to audiovisual forms for musical improvisation. It extends the performative practice of the musical looper, which enables musicians to record and play back multiple layers of sound in real-time. With the multimodal looper, we explore music cognition from an embodied perspective, considering cross-modal correspondences in order to achieve an intuitive method suitable for collaboration. Our goal is to create a live-looping system with multimodal objects that are gesturally activated and visually represented. Our first prototype focused on the essential modules of a live-looping system: for gesture recognition, we used a depth camera and a decision tree; for visuals that correspond to distinct categories of sounds, e.g., sustained, iterative, and impulsive, we employed procedural generation techniques such as animated noise, feedback loops, instancing, and various mathematical operations. The system’s effectiveness in establishing cross-modal correspondences for a multisensory experience was evaluated through user testing.