An Interactive Garment for Orchestra Conducting: IoT-enabled Textile & Machine Learning to Direct Musical Performance
Berit Greinke, Giorgia Petri, Pauline Vierne, Paul Bießmann, Alexandra Börner, Kaspar Schleiser, Emmanuel Baccelli, Claas Krause, Christopher Verworner, Felix Bießmann · 2021
We present an overview and initial results from a project bringing together orchestra conducting, e-textile material studies, costume tailoring, low power computing and machine learning (ML). We describe a wearable interactive system comprising of textile sensors embedded into a suit, low-power transmission and gesture recognition using creative computing tools. We introduce first observations made during the semi-participatory approach, which placed the conductor’s movements and personal performative expressiveness at the centre for technical and conceptual development. The project is a two-month collaboration between the Verworner-Krause Kammerorchester (VKKO), technical and design researchers, currently still running. Preliminary analyses of the data recorded while the conductor is wearing the prototype demonstrate that the developed system can be used to robustly decode a large number of conducting and performative movements. In particular the user interface of the ML system is designed such that the training of the algorithms can be intuitively controlled by the conductor, in sync with the MIDI clock.