Audio Concept Ranking for Video Event Detection on User-Generated Content
Benjamin Elizalde, Mirco Ravanelli, Gerald H Friedland · 2013
Video event detection on user-generated content (UGC) aims to find videos that show an observable event such as a wed-ding ceremony or birthday party rather than an object, such as a wedding dress, or an audio concept, such as music, speech or clapping. Different events are better described by different con-cepts. Therefore, proper audio concept classification enhances the search for acoustic cues in this challenge. However, audio concepts for training are typically chosen and annotated by hu-mans and are not necessarily relevant to a specific event or the distinguishing factor for a particular event. A typical ad-hoc annotation process ignores the complex characteristics of UGC audio, such as concept ambiguities, overlap, and duration. This paper presents a methodology to rank audio concepts based on relevance to the events and contribution to the ability to dis-criminate. A ranking measure guides an automatic selection of concepts in order to improve audio concept classification with the goal to improve video event detection. The ranking aids to determine and select the most relevant concepts for each event, to discard meaningless concepts, and to combine ambiguous sounds to enhance a concept, thereby suggesting a focus for an-notation and a better understanding of the UGC audio. Experi-ments show an improvement of the audio concepts mean classi-fication accuracy per frame as well as a better-defined diagonal in the confusion matrix and a higher relevance score. In terms of accuracy, the selection of top 40 audio concepts using our methodology outperforms the highest-accuracy-based selection by a relative 17.56 % and a frame-frequency-based selection by 5.74%. In terms of relevance to the events, the ranking-based selection provided the highest score. Index Terms: event detection, audio concept, user generated content, acoustic video processing.