Ranking and Extraction of Relevant Single Words in Text
João Ventura, Joaquim Ferreira da Silv · InTech eBooks · 2008
The process of extraction of relevant unigrams and n-grams is an area of great applicability. The most flagrant examples are associated, somehow, with the classification of documents. For instance, current search engines would benefit from having unigram and multiword extractors instead of returning results merely based on the occurrence of terms as they do nowadays. Also, applications like grouping and indexing of documents are also great candidates to benefit from this kind of extractors. However, the extraction of unigrams has been an almost ignored area by the scientific community. As it was mentioned before, to leave out unigrams in a process of extraction impoverishes the final results. The few approaches existent today suffer, however, a few problems. Essentially, they harm severely the frequent relevant words, when they are, as seen, pretty descriptive of the general topics of texts. On the other hand, all existent approaches are only capable of creating relevance rankings, which may be restrictive for some kind of applications like the characterization of keywords of documents. In this chapter we have presented two new metrics to evaluate words that are at the same time, language, frequency and context independent. Score measure is capable of improving results for very frequent words, while SPQ, besides that, has good results for Portuguese and Spanish (or other latin-descendent languages) documents. About the unigram extractor also presented in this chapter (Islands method), it allows to extract, with good results of efficiency, relevant unigrams from the relevance rankings. By the fact that any relevance rank can be used, this method is then metric independent. At last, we've presented the syllable method that can work as well as an isolated metric or with another metric. It has been seen that its results are encouraging. Although we have encouraging results, there can be, however, some improvements or further research following the sequence of this work. There is an interest in increasing even more the efficiency of all the methods, also increasing the values of Precision and Recall of the Islands method, arrange mechanisms to associate singular and plural terms and using synonyms, and, mostly, proceed with further research in the syllable area, a very promising area.