From Multimedia Logs to Personal Chronicles
Hyungik Oh, Ramesh Chandra Jain · 2017
Multimodal data streams are essential for analyzing personal life, environmental conditions, and social situations. Since these data streams have different granularities and semantics, the semantic gap becomes even more formidable. To make sense of all the multimodal correlated streams we must first synchronize them in the context of the application, and then analyze them to extract meaningful information. In this paper, we consider the problem of modeling an individual by using daily activity in order to understand their health and behavior. The first step is to correlate diverse data streams with atomic-interval, and segment a person's day into her daily activities. We collect the diverse data streams from the person's smartphone to classify every atomic-interval into a daily activity. Next, we use an interval growing technique for determining daily-activity-intervals and their attributes. Then, these daily-activity-intervals are labeled as the daily activities by using Bagging Formal Concept Analysis (BFCA). Finally, we build a personal chronicle, which is a person's time-ordered list of daily activities. This personal chronicle can then be used to model the person using learning techniques applied to daily activities in the chronicle and relating them to biomedical or behavioral signals. We present the results for this daily activity segmentation and recognition by using lifelogs of 23 participants.