Real time direction estimation for pointing interactions using a depth sensor and a nine axis inertial motion unit
Shome S. Das · 2020
Pointing devices are omnipresent in human-AI interaction scenarios. Existing 3D pointing devices like virtual reality(VR) wands, joystick, force balls, and track-pads are either cumbersome or restricted to operate on tabletop like surfaces. There exist gesture-based approaches for 3D pointing that use inertial motion units(IMU) based or computer vision based techniques. Existing IMU-based techniques are suitable only for pointing on fixed surfaces like big displays. Computer vision based methods suffer from issues like incorrect hand detection due to overlap with other body parts, restricted region of operation due to the use of multiple cameras and non real-time operation. To overcome the issues faced by existing commercial devices and techniques we propose index finger based 3D pointing direction estimation(hereafter known as PDE) technique that uses a combination of nine-axis IMU and depth data from a RGB-D sensor. The proposed method works in real-time and is highly robust to variation in the orientation and the depth of the hand w.r.t. the camera. The proposed method is suitable for personal usage applications that require high accuracy like computer games, VR, home automation etc. We show that our technique can be easily incorporated in existing hardware. We provide a few examples of use cases to enable readers to apply the proposed PDE technique to their applications.