Thin slices of head movements during problem solving reveal level of difficulty
Bart Joosten, M.A.A. van Amelsvoort, Emiel Krahmer, Eric O. Postma · Research portal (Tilburg University) · 2011
The aim of this paper is to study the feasibility of automatically reading the mental states from visual nonverbal expressions. In a recent study performed in our behavioral lab [1], we presented 57 children (second and fifth grade) with easy and hard arithmetic problems. We recorded videos of their non-verbal behaviors immediately following the presentation of the problems. Subsequently, 31 adults rated soundless versions of the 114 video fragments. The raters assessed the children’s nonverbal reactions to easy problems as reflecting a low perceived difficulty and their reactions to hard problems as indicating a high perceived difficulty. The human assessments of the soundless video fragments suggests that the non-verbal expressions of the children reveal information about their mental state with respect to the experienced level of difficulty of the arithmetic problem. State-of-the-art facial expression recognition methods may be able to capture such non-verbal information, thereby confirming or complementing the human assessments. In order to address whether this is the case, we performed a computational analysis of 110 of our fragments (4 fragments were unusable) using the Active Appearance Models (AAM) method [2, 4]. This method requires the manual specification of a grid of landmarks for a few representative frames of the video fragments. For facial expression analysis, the landmarks are placed at facial locations whose positions and appearances are of relevance for expressions. Once representative grids are created for a single video fragment, the AAM method uses these to automatically create grids for the rest of the frames in the fragment.