Computer selection of keywords using word‐frequency analysis
John M. Carroll, Robert Roeloffs · American Documentation · 1969
Abstract A statistically based method for automatically identifying keywords in machine‐readable text has been developed which produces keyword lists that agree better with composite lists produced by panels of human indexers than do lists produced by five statistical criteria previously suggested, and also better than lists produced by most of the individual panel members. The method makes use of both the in‐document word‐occurrence frequency and the in‐corpus relative occurrence frequency as measures of word importance. Each statistical criterion was compared with the performance of human indexers by the use of rank correlation statistics. The simple word count was found to be superior to the other four previously suggested criteria—all of which made use of the in‐corpus relative occurrence frequency. The tests were conducted over 19 documents dealing with the subject of Information Science, a total of over 66,000 word occurrences. Seventeen indexers representing eight different information centers participated in experiments.