Finding images expressing emotional, affective, abstract, and symbolic meanings
JungWon Yoon · Proceedings of the American Society for Information Science and Technology · 2007
Development of digital technology has spurred interest in accessing and utilizing digital images in a variety of fields, including education, entertainment, commercial industry, medicine, art, criminology, and so on. As a result, a number of image retrieval systems have become available via web search engines, digital libraries, and electronic image databases. However, despite this continuous attention to image retrieval, there is still dissonance between retrieved image documents and image searchers' needs. One reason for this dissonance is that current image retrieval approaches provide access to image documents mainly through denotative messages. Image documents inherently convey not only denotative messages but also connotative messages (Barthes, 1964/1977; Panofsky, 1962; Shatford, 1986), and people frequently seek images that express connotative messages (Keister, 1994; Jörgensen, 2003; O'Connor, O'Connor & Abbas, 1999). Therefore, connotative messages should be accessible to achieve more meaningful retrieval results; however, the notion that connotative messages are subjective and open to individual interpretation (Greisdorf & O'Connor, 2002) makes it difficult to implement an image retrieval system that supports connotative messages. The primary focus of this study is to verify a new approach designed to enhance accessibility to connotative messages during the image search process. This approach utilizes an image searcher's connotative reactions, which are quantitatively measured. The approach follows the lead of two lines of previous research. First, since connotative message of an image is dependent on not only the image document itself but also the relationship between the image document and the image viewer, there are researchers who have attempted to makes use of users' reactions as an image access mechanism (Brown & Hidderley, 1996; Jörgensen, 1995; O'Connor, 1996; O'Connor, O'Connor & Abbas, 1999). Based on their suggestions, the approach is designed to utilize users' connotative reactions. Second, the previous studies mostly depend on users' verbal descriptions for gathering users' reactions. Verbal descriptions are a traditional and convenient way to express reactions, but words have some limitations in articulating the connotative reactions to an image. In this sense, this study adopts quantitative measurements for gathering users' reactions. Two broadly used quantitative measurements, the Semantic Differential (SD) (Osgood, Suci, & Tannenbaum, 1957) and the Self-Assessment Manikin (SAM) (Lang, 1980), are considered as tools for gathering users' reactions. One advantage of using these quantitative measurements is that they can reveal complex connotative meanings in the reduced dimensions, and this distinctive feature is considered appropriate for integrating users' reactions into an image retrieval system. RQ1: How are the connotative meanings of an image represented through two quantitative tools, the SD and the SAM? RQ2: To what extent are there differences in connotative representations between the SD and the SAM? RQ3: To what extent are there correlations between connotative representation obtained by the SD and the SAM and connotative perceptions of image searchers? The findings of this study will reveal the implications for designing an image retrieval system that improves connotative message accessibility. This study's results will offer practical and fundamental data for designing and implementing an image retrieval system supporting connotative meanings. Additionally, the outcome of this study will be applicable to information retrieval systems containing multimedia information. The digital environment makes multimedia information a major resource for communicating and conveying messages. One of the features that are distinct from traditional text documents handled in the library community is that connotative message is critical to multimedia information, such as music, sound, moving images, etc. Although this study will focus on image documents, the framework and methodology adopted in this study can be applied to the diverse formats present in multimedia collections in digital libraries. Finally, as a theoretical implication, this study will explore connotative messages by utilizing quantitative measures, which has not yet been attempted. Therefore, the outcomes obtained from this new approach will broaden researchers' understanding of connotative messages in terms of information retrieval and user behaviors. This study will use quantitative methods, supplemented with qualitative information, to investigate three research questions. The survey questionnaire will be administered to approximately one hundred undergraduate/graduate students at the University of South Florida, based on the convenience sampling method. As mentioned above, the Semantic Differential and the self-Assessment Manikin will be used for quantitatively measuring users' connotative reactions towards an image. The instruments for the SD and the SAM are developed according to standardized guidelines suggested by previous researchers (Heise, 1970; Osgood, Suci, & Tannenbaym, 1957; Lang, Bradley, & Cuthbert, 2005). Participants will be shown a set of 30 color photographs which are selected from a photograph collection published on a CD with a book by O'Connor and Wyatt (2004). For each image, participants will be asked to rate an image using the SD and the SAM. Research question 1 & 2 will be explored by analyzing and comparing quantitative results of the SD and the SAM. Research question 3 is designed to examine correlations between connotative representation obtained by quantitative measuring tools and connotative perceptions of image searchers. Although there is no standardized tool for examining people's perception of an image, many information scientists have investigated people's image perception through three tasks: description, sorting and searching. This study also adopts these three tasks for examining people's connotative perception. The preliminary study was performed to investigate a possible effectiveness of a quantitative measure as an image retrieval mechanism. The preliminary study used the SD, because the SD has been verified in many previous studies and the SAM was devised as a simpler alternative of the SD. A total of 25 undergraduate/graduate students at the University of North Texas served as participants. The participants, who were assigned to a task, were asked to describe an image that they were expecting for achieving the given task and to indicate their expected images using the SD scales. Then, the participants were shown to the five images which were relevant to the given task and asked to indicate the appropriateness of each image (Figure 1) as well as to rate each image using the SD scales. Two dimensional plot shown in Figure 2 is a analysis result obtained from participants' reactions via the SD. According to the results, it was found that the expected image and the Images 4 and 5, which were judged as appropriate, were closely located. That is, it is an encouraging result showing the plausibility of the utilization of the SD in image retrieval processing through connecting expected image and relevant images, although the data analysis is based on a small sample size. Appropriateness of five images. Two dimensional plot of an expected image and five potentially relevant images