Real-Time Hand Tracking and Trajectory Gesture Recognition
Mohd Shahrimie Mohd Asaari, Ooi Shin Yi, Sami Abdulla Mohsen Saleh, Mohamad Khairi Ishak · 2025
With the advent of modern technologies, traditional input devices such as the mouse, keyboard, and remote control are becoming obsolete due to their lack of flexibility. In today's society, humans primarily communicate with computers using body language or voice commands, which are now widely integrated into most electronic devices. Hand gestures, in particular, are highly effective in human-computer interaction due to their natural expressiveness. However, vision-based hand gesture recognition systems face challenges such as complex backgrounds, illumination variations, and other environmental factors. Additionally, reliably detecting and tracking hands to extract trajectory information from video scenes remains a difficult task due to the diverse appearances of human hands, including variations in hand shapes, skin colors, illuminations, orientations, and scales in color images. Distinguishing between meaningful and meaningless motion trajectories further complicates dynamic hand gesture recognition. To address these challenges, a real-time hand tracking and gesture recognition system is proposed. This project implements real-time hand tracking and landmark estimation using Python, OpenCV, and MediaPipe. Hand trajectory gesture recognition is then achieved using a customized Convolutional Neural Network (CNN) with three convolutional layers, one flatten layer, and two fully connected layers. The network is built and trained using Matlab and trained model is exported to python using scipy.io module for model deployment. The model was trained on the MNIST dataset, which consists of 10 numeric gestures (0–9) with an overall testing accuracy of 97. 21%. When implemented on trajectory-based gestures, the system yielded an accuracy of 87.1%.