A 3D Hand Joint Detection Network for Real-Time Hand Capture and Its Application in a Game of Moving Mountains
Su Xin-Yue, Xue Hao-Wei, Meili Wang · 2023
Virtua1 reality (VR) technology has become one of the research hotspots of technology to promote teaching because of its high immersion and good interactivity. Putting VR games into education can make learning more participatory and incentive. Nevertheless, the majority of current technologies and efforts to accomplish human-computer interaction necessitate additional hardware, which has certain limits. To solve these problems, realize real-time operation to provide direct feedback, and enhance the user’s sense of immersion and experience, we have developed a real-time hand capture algorithm using a monocular camera to perform more accurate user gesture recognition. In the algorithm, we use the backbone architecture of deep residual network ResNet-50 as a feature extractor. Through the combined training of 2D and 3D annotation data, we are able to effectively predict 2D posture and 3D spatial information and implement virtual content interaction. This method achieves real-time performance (90fps) and accuracy (95.6%) on existing datasets and outperforms existing methods in hand mesh/pose accuracy and hand image alignment. We built and implemented a virtual reality game based on the method proposed in this research, and then transplanted it onto the VR platform with an ecological setting. While providing users with an immersive experience, we also want to use virtual reality technology to teach, play, and promote traditional Chinese culture.