Motion Eavesdropper: Smartwatch-based Handwriting Recognition Using Deep Learning
Hao Jiang · 2019
This paper focuses on the real-life scenario that people are handwriting while wearing small mobile devices on their wrists. We explore the possibility of eavesdropping privacy-related information based on motion signals. To achieve this, we elaborately develop a new deep learning-based motion sensing framework with four major components, i.e., recorder, signal preprocessor, feature extractor and handwriting recognizer. First, we integrate a series of simple yet effective signal processing techniques to purify the sensory data to reflect the kinetic property of a handwriting motion. Then we take advantage of properties of Multimodal Convolutional Neural Network (MCNN) to extract abstract features. After that, a bidirectional Long Short-Term Memory (BLSTM) network is exploited to model temporal dynamics. Finally, we incorporate Connectionist Temporal Classification (CTC) algorithm to realize end-to-end handwriting recognition. We prototype our design using a commercial off-the-shelf smartwatch and carry out extensive experiments. The encouraging results reveal that our system can robustly achieve an average accuracy of 64% at character-level and 71.9% at word-level, and 56.6% accuracy rate for words unseen in the training set under certain conditions, which expose the danger of privacy disclosure in daily lives.