Multimodal Analysis and Modeling of Nonverbal Behaviors during Tutoring
Joseph F. Grafsgaard · 2014
Detecting learning-centered affective states is difficult, yet crucial for adapting most effectively to users. Within tutoring in particular, the combined context of student task actions and tutorial dialogue shape the student's affective experience. As we move toward detecting affect, we may also supplement the task and dialogue streams with rich sensor data. In studies of introductory computer programming tutoring, human tutors communicated with students through text-based interfaces. Manual and automated approaches were leveraged to annotate dialogue, task actions, facial movements, postural positions, and hand-to-face gestures. Prior investigations in this line of doctoral research identified associations between nonverbal behavior and learning-centered affect, such as engagement and frustration. Additionally, preliminary work used hidden Markov models to analyze sequences of affective tutorial interaction. Further work will address the sequential nature of the multimodal data. This line of research is expected to improve automated understanding of learning-centered affect, with particular insights into how affect unfolds from moment to moment during tutoring. This may result in systems that treat student affect not as transient states, but instead as interconnected links in a student's path toward learning.