An Efficient Real-Time Air Drumming Approach Using MediaPipe Hand Gesture Model
Pabasara Surasinghe, Pasan Herath, Kokul Thanikasalam · 2023
The drum is a popular musical instrument used all over the world in many music styles. Although many people are interested in playing it, factors like cost and space make it inaccessible to everyone. Air drumming is a form of mimicking drumming motions in the air without the use of actual drums, and several computer-based approaches are available to generate drumming sounds from air drumming motions. Many of the current computer-based air drumming approaches rely on expensive sensors, and their detection accuracies are considerably low. In this study, we have proposed an efficient and real-time air drumming approach that is able to produce the drumming sounds from hand gesture movements. We have collected hand images from a few individuals and then created a hand gesture image dataset. Then the MediaPipe hand gesture model is utilized to extract the hand landmark coordinates. In the next step of the proposed work, a classifier is trained using the hand landmark coordinates to recognize 12 gestures. Finally, the trained classifier is employed to recognize gesture motions and generate the corresponding drum sound and visual feedback for the user. The proposed model demonstrated a classification accuracy of 99.63% on test images and an overall real-time drum sound detection accuracy of 89.56%. The proposed work reduces the gap between an air drummer and an automated system and is capable of functioning in real-time.