Eating and Drinking Behavior Recognition Using Multimodal Fusion
Amit Karmakar, Masafumi Nishida, Masafumi Nishimura · 2023
The automatic recognition of eating and drinking behavior is very important for human health tracking. However, these behaviors are quite challenging to recognize automatically due to their sporadic nature. Previous studies have focused on the recognition of a few behaviors, whereas our work focuses on the recognition of the nine most common eating and drinking behaviors to provide more in-depth information. We used three cameras and three inertial measurement unit (IMU) sensors to collect data, and we used different fusion techniques to automatically recognize these consumption behaviors. Using a novel approach with a multimodal Multi-Stage Temporal Convolutional Network, we were able to achieve an accuracy of 83.03% with a reasonable F1 score for individual classes. Additionally, we compared the performance of our approach with conventional approaches for insights. Our work may be helpful to future research on dietary monitoring and activity tracking.