Gesture Recognition applied to Extended Reality: A Case Study of Online Meeting
Yi-Jing Chen, Huai‐Sheng Huang · 2024
With the rise of the Metaverse, Extended Reality (XR) has successfully amalgamated real-life and virtual worlds, offering users novel immersive experiences, and gradually finding applications across various domains like gaming, education, healthcare, and more. Given the outbreak of the COVID-19 coronavirus in 2019, government-imposed social distancing measures resulted in a significant increase in remote work and schooling, bringing virtual social interactions back into widespread attention. This allowed individuals using Head-mounted displays (HMDs) to engage in online interactions with others in the form of avatars. However, avatars may not necessarily express hand movements, making real-time gesture recognition particularly crucial in virtual social contexts. The objective of this study is to collect participant hand data in virtual social interactions through HMD and utilize the WaveXR plugin for gesture recognition. Additionally, to assess the effectiveness of the WaveXR plugin’s gesture recognition, we have established a gesture recognition model based on CNN. The experimental results indicate that compared to the WaveXR plugin method, the CNN-based gesture recognition approach has improved F1 score by over 60%. This endeavor aims to enhance user experiences in XR, aiding both parties in assessing emotional affinity.