Practical personalized activity recognition systems with wearable motion sensors
Qu Tang · 2021
The COVID-19 outbreak demonstrates the need to measure hand hygiene behaviors such as handwashing activities and face touching actions to prevent infectious disease spread. Wearable technologies and supervised learning algorithms that interpret the motion data the wearables can collect may be used to detect these behaviors automatically. The bulk of prior work in human activity recognition using data from wearable motion sensors has assumed the activities to be recognized have motion patterns of a similar nature that are roughly the same in duration and often involve repetitive motion (e.g., ambulation). Activities such as handwashing and face touching, however, are heterogeneous --- each has unique characteristics in their motion patterns. Proper handwashing includes a series of discrete steps, with each handwashing step action involving a few seconds of repetitive, periodic motion. Face touching actions, alternatively, typically consist of a start, middle, and end, with little, if any, periodic motion, and can last for either just a second or many seconds. To segment and recognize both types of activities reliably in free-living, I demonstrate a deep neural network called "H-Net" (a modification of the U-Net network used in computer vision) that supports input sequences of wearable accelerometer data. Using the H-Net overcomes some of the limitations of prior fixed-size, sliding-window methods in two ways: a) it allows arbitrary length for inputs and uses convolutional and recurrent blocks to capture patterns over both short and long-duration activities, and b) it uses an encoder-decoder architecture that allows the model to be used to segment activities in the input sequences. My solution uses person-tailored training data. I demonstrate why personalized training data is valuable, how such data might be obtained, how to automatically augment the amount of training data gathered to improve the performance of the H-Net. I report on the overall performance of a recognition system on the handwashing and face touching detection task on data collected from people using the system outside of the lab environment without researcher supervision. To validate the proposed approaches, I collected two hand hygiene datasets --- one collected using a proposed app-guided approach for collecting training data (17 participants), and the other collected from free-living using a wearable camera (2 participants). The proposed system outperforms window-based baseline models on these datasets, achieving an average F1-score of 0.82 over 16 classes (i.e., 12 handwashing steps, face touching, confounding classes of face touching and handwashing, and the Context class), which is better than the average F1-score of 0.73 with models using fixed-size sliding-window methods. The proposed method does not require the selection of an optimal window size; the performance of prior methods is highly dependent on window size. I explore the utility of collecting additional non-tailored training data and additional person-specific free-living training data, and I investigate the feasibility of using one wrist sensor instead of two. I conclude by discussing the opportunities and challenges of incorporating the H-Net handwashing and face touching recognition system into future real-time training applications.--Author's abstract