WindowSpace: A Web-Based XR Window Manager for Interacting with 2D Windows in Immersive 3D Space
Emeric Marguet, Marcel Borowski, Janus Bager Kristensen, Clemens Nylandsted Klokmose, Niklas Elmqvist · Proceedings of the ACM on Human-Computer Interaction · 2025
Extended Reality (XR) head-mounted displays (HMDs) coupled with standard mouse and keyboard hardware present a unique opportunity to rethink traditional desktop productivity. In this paper, we propose WindowSpace, an XR-based window manager for placing 2D windows in immersive 3D space while using a traditional mouse and keyboard. WindowSpace is a web-based system supporting consumer-level HMD-AR that can display native 2D windows from standard operating systems using a native desktop streaming server based on a standards-compliant optimized WebRTC protocol. To facilitate interacting with these native applications, we investigate the design space for effectively controlling the cursor between windows arranged in immersive 3D space. We propose three pointing techniques in our design space: an off-screen cursor (EtherXR), a head-tracking based pointer (Face&Jump), and a novel technique based on switching screens by extending the mouse trajectory (BallisticXR). Our empirical study revealed that EtherXR performs slightly better overall; however, BallisticXR showed better accuracy. Notably, Face&Jump yielded slower and less accurate performance, suggesting that full gaze assistance may be necessary to support this approach effectively.