Enhanced Context Recognition by Sensitivity Pruned Vocabularies
Rasmus Elsborg Madsen, Sigurdur Oli Sigurdsson, Lars Kai Hansen · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · 2004
Document categorization tasks using the "bag-ofwords" representation have been successful in instances [11].The relatively low dimensional bag-of-words form, is well suited for machine learning methods.The pattern recognition methods suffers though, from the well-known curse of dimensionality, since the number of input dimensions (words) usually supersedes the number of examples (documents).This high dimensional representation is also containing many inconsistent words, possessing little or no generalizable discriminative power, and should therefore be regarded as noise.Using all the words in the vocabulary is therefore resulting in reduced generalization performance of classifiers.We here study the effect of sensitivity based pruning of the bag-of-words representation.We consider neural network based sensitivity maps for determination of term relevancy, when pruning the vocabularies.With reduced vocabularies documents are classified using a latent semantic indexing representation and a probabilistic neural network classifier.Pruning the vocabularies to approximately 3% of the original size, we find consistent context recognition enhancement for two mid size data-sets for a range of training set sizes.We also study the applicability of the sensitivity measure for automated keyword generation.