CoboGesture: A Continuous Hand Gesture Dataset and Recognition Method for Human-Collaborative Robot Interaction
Phuong-Dung Nguyen, Van-Thang Tran, Duy-Khanh Ngo, Viet-Duc Le, Ha-Anh Nguyen, Thuy-Binh Nguyen, Hong-Quan Nguyen, Thi‐Lan Le · IEEE Access · 2026
The use of collaborative robots (aka cobots) - a type of robot designed to work alongside humans have recently increased. To create a seamless collaboration between human and cobot, hand gesture is the most widely used. In this paper, we focus on addressing the hand gesture recognition for human-cobot interaction. Recently, many models have been proposed for hand gesture recognition using various modalities, including RGB, depth, and skeleton data. However, the majority focus on recognizing isolated hand gestures on a segmented video sequence which limits their practical application. In this paper, we propose two methods for continuous hand gesture recognition, namely RGB-CoGes and SkeRGB-CoGes. The RGB-CoGes method relies solely on RGB image sequences, while SkeRGB-CoGes integrates both skeleton and RGB modalities. Both methods share a common training phase, in which an isolated gesture recognition model based on VideoMAE and VideoMAEv2 is trained on segmented sequences. During inference phase, RGB-CoGes employs a sliding window approach on continuous sequences to classify gestures within each window using the trained model. In contrast, SkeRGB-CoGes leverages the lightweight skeleton information to detect the start and end points of gestures, after which the same trained model is applied to the identified segments. Additionally, to evaluate and promote the research on continuous hand gesture recognition, a self-built dataset named CoboGesture containing 19 different hand gestures in the context of human-cobot interaction has been collected and fully annotated. Experiment results demonstrate the effectiveness of the proposed framework in enhancing gesture recognition performance for human-cobot interaction. Regarding continuous recognition, the proposed methods RGB-CoGes and SkeRGB-CoGes achieved frame-wise accuracies of 0.87 and 0.90, and overlap scores of 0.68 and 0.67, respectively, significantly outperforming the skeleton-based methods. Furthermore, we demonstrated the successful deployment of the proposed method on an edge device.