mmWave MIMO Radar-Based Handwriting Recognition With Lightweight Fingertip Tracking
Jyun-Jhih Lin, Yen-Hsiang Tseng, Wei-Chang Chen, Nan Wu, Tingting Zhang, Po‐Hsuan Tseng · IEEE Sensors Journal · 2024
Using wireless signals such as millimeter-wave (mmWave) multiple-input-multiple-output (MIMO) radar for human-computer interaction is advantageous in maintaining privacy, and it is not easy to be disturbed by light. In this article, we design a mmPoint fingertip tracking and integrate it with a contactless radar-based handwriting recognition (radarHR) system. The radar point cloud is used as the model input data and a multimedia machine-learning model application framework dimension–Mediapipe, is used as the trajectory labeling. Due to the sparse property of the fingertip, the proposed mmPoint encoder extracts point cloud features based on a space-filing curve (SFC) or PointNet and then uses long short-term memory (LSTM) to obtain temporal features, replacing the 3-D voxel model that consumes computing resources. To evaluate the similarity of the mmPoint fingertip trajectory to the handwriting trajectory, the performance of mmPoint fingertip tracking is conducted using a classification model training from a handwritten letter dataset-EMNIST. Compared with the 3-D voxel model, the SFC model is the most computationally efficient, with 289 times less computation and 20 times fewer model parameters. The PointNet model has an advantage in the tradeoff between accuracy and model calculation. The classification accuracy of numbers and letters for the PointNet model is 20% ahead of the voxel model. It is about 10% higher than SFC, consuming 14 times less computation and 2.5 times fewer parameters than the voxel model.