A Framework for Creative Experimentation in Extended Reality Using Face-Centered Spatial Relationships and On-Device Inference
Stavroula Zoi, George Caridakis · 2025
Extended Reality (XR) offers a powerful conceptual framework and a creative ecosystem for developing spatially aware content. While proprietary wearable devices establish structured spatial relationships through user and environmental tracking, complementary open, cross-platform approaches enable artists and content creators to explore novel, unconventional interactions using accessible devices. Such approaches enhance key aspects of the creative process, including continuous sketching, self-observation, freedom of movement, mental mapping, and dynamic content transformation, while maintaining privacy. This paper presents an experimental framework for creative sketching and exploration in XR, leveraging spatial relationships between the creator, the device, and the surrounding space through face-tracking and smartphone sensors. The framework is implemented using Google's MediaPipe, an open-source on-device machine-learning solution, and Unity. To demonstrate its potential, this study analyzes the arrangement of content 360° around the creator. Preliminary experimentations are presented and future research directions are discussed.