Multimodal Food Journaling
Hyungik Oh, Jonathan Nguyen, Soundarya Soundararajan, Ramesh Chandra Jain · 2018
A food journal is essential for improving health and well-being. However, journaling every meal is extremely difficult because it depends on user initiative and intervention. Current approaches to food journaling are both potentially inaccurate and tedious, causing people to abandon their journals very soon after they start. In this paper, we propose a proactive and reactive mechanism that can significantly reduce user initiative while still remaining highly accurate. We first suggest a novel eating moment recognition technique using heart rate and activity patterns to trigger a food journaling process in a proactive manner. We then begin the food journaling process via voice command which utilized natural language processing when logging meals, which increases the ease of reactive self-reporting. Lastly, we enhance the food journal by automatically assessing ecological moments of eating activity through our personal chronicle system. We verified the method from a feasibility study conducted with three people for three months in their day-to-day lives. Our approach is designed to be unobtrusive and practical by leveraging multi-modal sensor data through the most common device combination of a smartphone and wearable device.