Privacy-preserving Finger Movement Tracking U sing Acoustic Sensing Enhanced by Smartphone Case Mini-structures
Zhengkun Ye, Yan Wang, Yingying Chen · 2024
Traditional smartphone touchscreens often raise privacy concerns. We thus propose a novel system using low-cost smartphone cases for privacy-preserving finger activity sensing via passive acoustic signals from the back of the smartphone case. It leverages embedded mini-structures to regulate and enhance the acoustic signals gen-erated by finger activities on the case. We develop a multi-task learning framework with a multi-scale shared encoder and task-specific decoders to extract comprehensive acous-tic features of finger activities. During offline training, our system uses raw passive finger activity sound as input and camera supervision. A Siamese network is utilized to extract finger activity-specific feature files, eliminating the need for users to collect training data. Initial experimental evaluations validate the system's superior performance.