Evaluation of information-seeking performance in hypermedia digital libraries
Michail Salampasis, John Irving Tait, Chris Bloor · Interacting with Computers · 1998
Abstract Nowadays, we are witnessing the development of new information-seeking environments and applications such as hypermedia digital libraries. Information Retrieval (IR) is increasingly embedded in these environments and plays a cornerstone role. However, in hypermedia digital libraries IR is a part of a large and complex user-centred information-seeking environment. In particular, information seeking is also possible using non-analytical, opportunistic and intuitive browsing strategies. This paper discusses the particular evaluation problems posed by these current developments. Current methods based on Recall (R) and Precision (P) for evaluating IR are discussed, and their suitability for evaluating the performance of hypermedia digital libraries is examined. We argue that these evaluation methods cannot be directly applied, mainly because they do not measure the effectiveness of browsing strategies; the underlying notion of relevance ignores the highly interconnected nature of hypermedia information and misses the reality of how information seekers work in these environments. Therefore, we propose a new quantitative evaluation methodology, based on the structural analysis of hypermedia networks and the navigational and search state patterns of information seekers. Although the proposed methodology retains some of the characteristics (and criticisms) of R and P evaluations, it could be more suitable than them for measuring the performance of information-seeking environments where information seekers can utilize arbitrary mixtures of browsing and query-based searching strategies.