Action recognition algorithm from multimodal images for human-robot collaboration systems
Viacheslav Voronin, Marina M. Zhdanova, Evgeny A. Semenishchev, Alexander A. Zelensky, Nikolay Gapon, Yurii V. Ilyukhin · 2023
Automation of production processes using robots is a priority for developing many industrial enterprises. Human-machine interaction is a key component of such control infrastructure. The proposed algorithm is a four-stage procedure: (a) fusion information from multimodal sensors based on the quaternion model, (b) image preprocessing using a 3D Gabor filter, (c) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (d) classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes; for intensified motion, used the convolution with a bank of 3D arbitrarily oriented Gabor filters and calculating 3D local binary dense micro-block difference. A program was developed for transmitting information about the target points of the robot's movement through a virtual TCP / IP port and a script for working out the target points in the simulation environment. To test the effectiveness of the proposed algorithm, we simulate an action recognition system for Human-robot collaboration in the RoboGuide environment.