Coding Hand Movement Behavior and Gesture with NEUROGES Supported by Automatic Video Analysis
Oliver Schreer, Stefano Masneri, Harald Skomroch, Hedda Lausberg, Fraunhofer Heinrich · 2014
In many different research disciplines in humanities, very large multimodal corpora are being processed and analysed in order to solve quite a large variety of different research questions. Not only in the gesture community, but also in psycholinguistics and psychology, the way how human act with their hands and body is of relevance. To solve the different research questions, a detailed annotation of multimodal data (video and speech) is performed. This annotation is basically performed manually by human raters and it is usually a very time consuming process. In empirical analysis of hand movement behaviour and gesture, the coding system NEUROGES and the annotation tool ELAN are very useful tools in this research domain [1][2]. Although, the manual annotation process is supported very well, it is still a very time consuming process to get at least the basic assignment such as start and end of hand movements to perform the seemingly most simple step, i.e., to segment the ongoing flow of behaviour into natural units. We present an automatic video analysis tool that supports the researchers in this exhaustive exercise inspecting and analysing videos that substantially facilitates the segmentation and annotation processes. Many significant events in human body motion can be detected quite robust by modern video analysis tools and therefore speed up the annotation process by a serious amount. The NEUROGES coding system provides a seven steps algorithm to analyse hand movement behaviour including gestures. It consists of three modules progressing from segmenting and annotating behaviour according to Activation, Structure, and Focus (Module I) to analyzing the relation of the right and left hands concerning Contact and Formal Relation (Module II) to finally assessing the Function and Type of the movements (Module III). The developed video analysis tool is able to support the annotation in a fully automatic way and therefore can speed up the overall annotation process significantly.