The influence of recall feedback in information retrieval on user satisfaction and user behavior
E.C.M. Hoenkamp, H.C. van Vugt · eScholarship (California Digital Library) · 2001
The unprecedented scale-up of the World Wide Web, and the number of people relying on it for information, make it inevitable to reassess the validity of the traditional metrics for quality of information retrieval (IR). Of these, the most widely used metrics are recall and precision. Users can judge the precision of an information retrieval system by inspecting the retrieved documents. They cannot judge recall, however, which would involve inspecting the whole collection, thus obviating the IR system, and impossible in the case of WWW. How then, can we ascertain whether recall is a valid metric for the quality of an IR system as perceived by the end-user? In a carefully controlled experiment we presented users with a simulated web search engine. Besides the search results, the engine could give a (spurious) recall estimate, presented as a pie chart. We manipulated this recall feedback, and whether the information need was fulfilled with respect to quantification type (the number of documents requested). It seems that fulfillment is a better predictor of user satisfaction and behavior than precision and recall as used to evaluate IR systems. The results reported here also suggest that whereas recall may be a valid metric for designers and evaluators of IR systems, it may lack validity as a metric for search quality as perceived by the end-user.