A Labeling Support System for Participants’ Situated Assessments by Multi-modal Sensing
Ayumi Ohnishi, Kyosuke Kondo, Hiroshi Kato, Tsutomu Terada, Hideaki Kuzuoka, Yoshihiko Kubota, Hideyuki Suzuki, Masahiko Tsukamoto · 2021
This research aims to build a system that estimates assessment expressions to others by machine learning, collects them in real time, and feeds them back to the participants, and this study proposes two techniques that are necessary for this purpose. First, we propose a method to estimate the evaluative behavior to others, which is simplified to positive, negative, and neutral, from the facial images and head acceleration of participants in the conversation. The supervised machine learning algorithm requires ground-truth labeling. To analyze the expression of a large number of people, such as in a classroom, ground-truth labeling becomes a challenge in analysis and system construction. Second, to ease the labeling process we propose and implement a method to recommend the estimated labels on the labeling software. The accuracy of the proposed method is approximately 0.45 in F-value. Considering the time-series information, the proposed method recognizes major changes in positive evaluation behaviors. Presenting recommended labels on our labeling support system increase with time for the ground-truth labeling. However, the accuracy of labeling positive assessments improve in five out of six subjects.