Knock&Tap: Classification and Localization of Knock and Tap Gestures using Deep Sound Transfer Learning
Jae-Yeop Jeong, Jung-Hwa Kim, Ha-Yeong Yoon, Jin-Woo Jeong · 2021
Gesture interaction is considered one of the promising approaches to control smart devices. In this paper, we present Knock&Tap, an audio-based approach that can perform gesture classification and gesture localization using deep transfer learning. Knock&Tap consists of a single 4-microphone array to record the sound of the user's knocking and tapping gestures and a wood/glass panel for knocking and tapping. Knock&Tap can be used in a situation or environment where vision-based gesture recognition is impossible due to the lighting condition or camera installation issue. Various experiments were conducted to validate the feasibility of Knock&Tap with 7 gesture types on both wood and glass panels. Our experimental results show that Knock&Tap predicts the gesture type and location with an accuracy of up to 97.24% and 92.05%, respectively.