Facial Gesture Classification with Few-shot Learning Using Limited Calibration Data from Photo-reflective Sensors on Smart Eyewear
Katsutoshi Masai, Maki Sugimoto, Brian Kenji Iwana · 2024
This study investigates smart eyewear for facial gesture classification with low user calibration costs.The smart eyewear is equipped with low-cost, comfortable, energy-efficient photo-reflective sensors, which can detect changes in facial muscle movements.Although the sensor output is useful for facial gesture classification, individual user calibration is considered necessary.Moreover, re-calibration is required whenever the user's wearing position changes.Therefore, reducing the calibration cost is crucial for the wider applicability of the eyewear.To address this issue, we propose a few-shot domain adaptation approach using Convolutional Neural Networks (CNN).We evaluate the accuracy of classifying eight gestures with data augmentation and a supervised contrastive loss.Data augmentation is employed to make the model more robust to noise, while the supervised contrastive loss is introduced to learn user-invariant features.Our approach with data augmentation achieves robust gesture classification, with an average accuracy of 93.46% (SD = 8.34%) for three emotion-related gestures without user-specific data.Furthermore, for user-independent training, we demonstrated that using few-shot learning with pre-trained models and only four repetitions of calibration data per gesture achieved a practical accuracy of 91.38% (SD = 6.03%), showing that a small amount of user-specific data is sufficient for the accurate classification.Also, it works under different wearing conditions, achieving an accuracy of 90.19% (SD = 3.56%).These results illustrate the potential of our method to improve the practicality of smart eyewear for facial expression recognition in cases of limited user data, making it more accessible and user-friendly.