Autolume-Live: Turning GANs into a Live VJing tool
Jonas Kraasch, Philippe Pasquier · 2022
Creative Artificial Intelligence is increasingly used to generate static and animat- ed visuals. While there are a host of systems to generate images, videos and music videos, there is a lack of real time video synthesizers for live music per- formances. To address this gap, we propose Autolume-Live, the first GAN-based live VJing-system for controllable video generation. By analysing the needs of VJs and current Deep Learning-based audio-visual systems, we developed a live video synthesiser that extracts musical features, such as the amplitude, pitch and onset strength, and allows mapping these to generate trajectories in the frame latent space. These frames can further be manipulated through a MIDI In- terface which allows the user to improvise, manipulate the Neural Network and adjust the output to accompany live music performances in a co-creative way.