An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval.
Hien Phuong Nhat Nguyen, Eugene Santos · 2007
Abstract This paper reports our evaluation of the ac-curacy of capturing a user’s intent in an information-seeking task. Specifically, we would like to assess how accurately a user’s short-term goals, methods, and con-text in an information seeking task have been captured. Our method is to compare a machine-generated model against a human-generated model with 5 users using the CACM collection. Our results demonstrate a good coverage of human-generated user models by machine-generated models in terms of short-term goals, methods and context. The similarity between a real user and our approach in terms of context agrees with the similarities of human-generated ontologies using the same metrics in existing studies from the ontology community. Fur-thermore, our results show that the similarities between a human and machine in terms of short term goals and methods, which affect the relevancy assessment process, agree with the overlap among people while assessing rel-evancy documents in the existing study from the infor-mation retrieval community.