XR Jam: Design Considerations for Music Networking with AI in XR
Torin Hopkins, Suibi Che Chuan Weng, Rishi Vanukuru, Sasha Novack, Chad Tobin, Emma Wenzel, Amy Ulinski Banic, Mark D. Gross, Ellen Yi-Luen Do · 2024
The XR Jam project presents an integrated system designed to enhance musical interaction and spatial awareness in Extended Reality (XR) environments. Leveraging AI-driven Virtual Musical Agents (AIVMs), the platform elevates traditional musical experiences. Built on an XR framework, the system pushes conventional boundaries by enabling novel forms of interaction, such as object manipulation and social drumming with AIVMs and remote players. Furthermore, avatar-based interfaces help overcome social inhibitions, offering players a unique, uninhibited interactive experience. Pilot studies revealed the possibility for effective use of haptics in spatial localization. While haptic feedback enhanced spatial orientation in specific scenarios, front and back directions remained challenging to discern. The system promotes musical interaction, social engagement, and spatial orientation, although further refinements are needed for synchronization and player experience. Overall, this research opens new avenues for immersive musical experiences through a fusion of AI, XR, and haptic technologies.