Expertise, search behavior, and search performance of engineering users
Xiangmin Zhang · Proceedings of the American Society for Information Science and Technology · 2003
A fundamental challenge to user modeling for information retrieval (IR) systems is to decide what user knowledge should be included in user models. Such knowledge should be able to enable an IR system to effectively predict the user's search behavior/performance and to effectively personalize the search process based on the user's model. A better understanding of user characteristics and user behavior will provide a basis on which individualized IR systems can be designed. This study investigates the roles of users' subject domain knowledge and IR knowledge in their search behavior and performance. User studies in IR have generated a rich literature. For example, Borgman (1989) examined individual differences in information retrieval in terms of personal characteristics, technical aptitudes, and academic orientation and concluded that these factors were interrelated. Allen (1991) found that there was a relationship between the level of domain (topic) knowledge and recall in searches in an online library catalog. The study revealed that high-knowledge users in the subject area of Voyager 2 exploration of Neptune had a greater familiarity with the vocabulary of the topic. Yee (1993) compared search strategies of graduate students in library science and education when searching in both in their own domain and the opposite domain. For the students in education there were no differences in search behavior when conducting searches on familiar versus unfamiliar topics. The library science students took more time to prepare off-line the search on education administration and they spent more time to evaluate the results in this unfamiliar field as opposed to the time they spent to conduct the search in their own field. The library science students used more thesaurus terms and more synonyms in the search on education. Marchionini, Dwiggins, Katz, & Lin (1993) compared domain experts with intermediary search experts. They revealed that domain experts were contentdriven, focusing on the answers to the search questions and had clear expectations for the answer to be found while search experts were problem-driven focusing on the problem statement and the query formulation. Kiestra, Stokmans, & Kamphuis (1994) found that the domain knowledge had a significant effect only on search time. Bhavnani (2002) revealed the existence of domain-specific search knowledge in healthcare and online shopping, which consisted of goal sequencing strategies and important URLs for each goal. The results from these studies seem to be inconsistent, and most of these studies focused only on the differences among/between different types of domains, e.g., search knowledge and domain knowledge. How the amount of the knowledge in a domain affects a user's performance/behavior is rarely investigated. Additional research is needed to determine the roles of the factors in search performance and behavior. Users' expertise, both in subject domain related to the research topics provided by a class instructor, and in IR systems; The number of queries used for a research question/topic in order to obtain satisfactory results; The search terms used in queries; and Successfulness of the user's search results in terms of the number of relevant documents identified for each research topic. The subject field chosen for this study is Heat and Thermodynamics in engineering. Twenty-four Wayne State University engineering students participated in the study. These students were assumed to have different levels of domain knowledge in relation to the field chosen for this study. The study used the following instruments for collecting data: IR system: COMPENDEX, one of the most frequently used engineering database which is available through the Axiom's web-based database service. Thesaurus: Engineering Information Thesaurus, 2nd edition (1995), for eliciting and representing the subjects' domain knowledge. About 200 terms from the Heat and Thermodynamics section of the EI Thesaurus were converted into five-point scales for the participants to rate their level of familiarity with each term. IR concepts and attributes: which were generated by using the repertory grid technique, for eliciting and representing the subjects' IR knowledge. These concepts and attributes are also transformed into 5-point scales for the subjects to rate on. User questionnaire: To obtaining subjects' basic demographic data, as well as to assess their search experience. Search questions: Three research topics for the engineering class projects. They were generated by the instructor of the class. Post-search form: To review the search processes, e.g., how relevance judgments were made, how search queries were refined, and the subject's general satisfaction with the results. Computer logs: Search history and results are printed and saved in a computer file. (Pre-search) fill out the user questionnaire and complete the Thesaurus term rating form and IR concept rating form. To avoid the possible influence of the thesaurus terms on searching tasks, the thesaurus term rating was done no less than 48 hours before the search session; (Search) perform searches on the 3 search questions. Participants assessed the relevance of their search results; (Post-search) fill out the post-search evaluation form to indicate level of satisfaction with search results, which immediately followed the search session. The result from a pilot study, which involved three participants, demonstrated that higher levels of domain knowledge tend to have fewer searches and tend to have more relevant documents (Zhang, Anghelescu & McMinn, 2001). However, these results are very preliminary and can only be taken as samples. The user data collection has completed and the results of the formal data analyses will be presented at the conference. This research was supported partially by a Wayne State University small research grant.