Multimedia web information fusion and analysis
Jiang Tao · 2008
Information on the World Wide Web appears in diverse forms, including text, image, audio, and video.Presented with a wide range of information, an information user often takes great effort to correlate and track online information related to specific topics of interest.Fusion of multimedia information in a unified framework is thus needed for efficiently understanding and further analyzing the semantically related information.This thesis addresses the problem of multimedia web information fusion and analysis by presenting an approach for modelling multimedia information in a unified semantic framework, based on which cross-media information analysis and mining is realized.As multimedia data are heterogeneous in their contents and formats, we employ a strategy for multimedia information fusion based on semantics of the data.Specifically, we develop two methods, one using a statistical vague transformation technique and the other employing a self-organizing neural network, to associate web images with related surrounding texts, based on which the semantics of the media objects can be extracted.Our experiments show that the proposed methods can identify associated image and text pairs with good accuracy and outperform a state-of-the-art method for image annotation using a statistical relevance model.To support cross-media analysis, this thesis develops a semantic representation schema, that combines MPEG-7 multimedia description, RDF language specification, and conceptual graph based knowledge representation techniques for modelling multimedia information.In addition, we develop a semantic metadata extraction algorithm utilizing a myriad of natural language processing (NLP) techniques to automatically extract concepts and relations from text contents.The extracted concepts are formally represented as bags of WordNet senses, based on which an incremental clustering approach i