Trading Virtual Objects Quality for AI Performance in Mobile Augmented Reality Apps
Niloofar Didar, Marco Brocanelli · 2024
Typical Mobile Augmented Reality (MAR) apps include several compute-intensive tasks for rendering virtual objects (AR tasks) and analyzing the real environment through AI model inference (AI tasks). Unfortunately, most of the existing research work overlooks the computational concurrency of AR and AI tasks. In fact, increasing the total triangle count of virtual objects may improve virtual object quality but reduce AI inference performance (e.g., latency). In this paper, we design MIR, a framework for MAR apps that dynamically regulates the on-screen triangle count to trade off between the performance of AR and AI tasks, leveraging locally-trained linear performance models and an approximation algorithm. We have implemented MIR on Android and tested it on real smartphones and users against several baselines to prove it helps improve performance with minimal resource usage overhead.