An experiment stemming non-traditional text
Mário A. Nascimento, A.C.R. Da Cunha · 2002
Stemming is a technique which aims to extract common suffixes of words. Thus, words which are literally different but have a common stem, may be abstracted by their common stem. The underlying goal when using a stemming technique is to improve recall, at the possible expense of precision loss. A well known technique for stemming text is M.F. Porter's (1980) algorithm, which is based on a set of rules extracted from the English language. We argue that such an algorithm it is not efficient for non traditional texts, e.g., one made up mainly of medical terms. We thus investigate the use of a technique, called Peak-and-Plateau, which is based on tries, and compare it to Porter's algorithm. Our experiments have shown that using Porter's algorithm or none at all makes no difference as far as precision and recall goes. On the other hand using the Peak-and-Plateau technique we improved recall by about 15% and decreased precision by an average of 40%. Moreover it compressed the original text by 40% and the invented file by 45%.