Characterizing music in multimedia contexts via user scene descriptions
Cynthia C. S. Liem, Martha Moon Larson, Alan Hanjalić · 2013
Music frequently occurs as an important rein- forcing and meaning-creating element in multimodal human experiences. This way, cross-modal connotative associations are established, which are actively exploited in professional multimedia productions. A lay user who wants to use music in a similar way may have a result in mind, but may lack the right musical vocabulary to express the corresponding information need. However, if the connotative associations between music and visual narrative are strong enough, char- acterizations of music in terms of a narrative multimedia context can be envisioned. In this article, we present the out- comes of a user study considering this problem. Through a surveyforwhichrespondentswererecruitedviacrowdsourc- ingmethods,wesoliciteddescriptionsofcinematicsituations for which fragments of royalty-free production music would be suitable soundtracks. As we will show, these descriptions can reliably be recognized by other respondents as belong- ing to the music fragments that triggered them. We do not fix any description vocabulary beforehand, but rather give respondents a lot of freedom to express their associations. From these free descriptions, common narrative elements emerge that can be generalized in terms of event structure. The insights gained this way can be used to inform new con- ceptual foundations for supervised methods, and to provide new perspectives on meaningful and multimedia context- aware querying, retrieval and analysis.