Temporal precision and reliability of audience response system based annotation
Jens Edlund, Samer Al Moubayed, Christina Tånnander, Joakim Gustafson · 2013
Abstract. Manual annotators are often used to label human interaction data. This is associated with high costs and high time consumption, which is one reason annotation by crowd sourcing is increasing. But in crowd sourcing, control over the conditions is largely lost. To get increased throughput with a higher measure of experimental control, we suggest borrowing from the Audience Response Systems used extensively in the film and television industries. We present (a) a cost-efficient setup for rapid, plenary annotation of human spoken interaction data; and (b) a study that quantifies the temporal precision and reliability of annotations made with the system in the auditory perception domain.