On relevance weights with little relevance information
Stephen E. Robertson, Steve Walker · 1997
The relationsh~p between the Robertson/Sptxck Jones relevancxi mighting formula and the Croft/Harper version for no relevanee information is d~ussed.A method of avoiding the negative weighta sometimes implied by the Cr&/Harper version is propowx$ which turna out to involve a return to the original Sparck Jones inverse collection frequency weight.The paper then goes on to propose a new way of using small amounts of relevanee information in the e9tim* tion of relevance weiglh Some experiments using TREC data are reported.1 A short hiiry This paper concerns the weighting of search terms as a searching mechanism, when there may be some relevance judgments on a f~documents in relation to the query or information need for which the search is being made.In this section, some elements of the history of relevance weighting are review@ as background to the present paper.Only those aspects relevant to the present discussion are covered.1.1 Inversecollectionfrequencyweighting Weighting terms accordiig to the number of documents in which they occur or to which they are assigned was d-vered empirically by Sparck Jones to be w ef%ctive search device [1].Frequent terms (that is, those that occur in many @cuments) are not in general good diecriminatm, and should be given low Weighs thus the weight should be inverdy related to the frequency.Such weighting is known as inverse document kquency (IDF) or inverse collection frequency.The usual formula is N w= logn where w is the weight to be assigned to the term, N is the size of (number of documents in) the collection and n is the number of documents in which the term occurs.Pmnkion to snakadigitalhrd mpicaofall of pad ofIhis materialfw pcraonal orClawoonlusc is_ withouifee providedthatthe copies arcnotmadaordkrimcd fw promor commercial advantage, the copy-n~tiistiti edtipbtiti dh-e, ddak givmthatcopyri