ℱsign-Net: Depth Sensor Aggregated Frame-Based Fourier Network for Sign Word Recognition
Sunusi Bala Abdullahi, Kosin Chamnongthai, Lubna A. Gabralla, Haruna Chiroma · IEEE Sensors Journal · 2024
Hand tracking is a challenging problem during hand gesture recognition due to abnormal hand patterns across depth signs and errors between normal pixels and backgrounds. In this article, we propose$\mathscr {F}$sign-Net: Fourier pixelwise approach based on the Fourier convolution neural network (FCNN) and time-distributed-based bidirectional long short-term memory (BiLSTM). FCNNs have been widely researched, reaching the state-of-the-art (SOTA) performance on spatial recognition tasks. However, it is still difficult for the Fourier model to learn the temporal patterns due to the chaotic nature of the hand motion data.$\mathscr {F}$sign-Net aggregates time-based information from spatial and temporal modules to a given Fourier convolution in three stages: 1) each depth frame is regarded as a window, and it is selected so that the aggregated sums of the pixels across the hand joints of the selected window are aligned; 2) a truncated pooling is applied that summarizes the generated featured map of the Fourier convolution to avoid over-fitting; and 3) the long-term temporal dependencies among pixels for the Fourier convolution are captured using the shared time-based BiLSTM layers. This allows the proposed model to learn hand patterns that are temporally oriented. Finally, the proposed$\mathscr {F}$net-Sign is evaluated on depth sign language public datasets and demonstrates the SOTA performance. Simulation results proved that improved Fourier features are good features for the proposed hand-tracking approach.