An evaluation component for categorization systems
Peter Gerstl, Ulrich Hofmann, Allen L. Lang · WIT transactions on information and communication technologies · 2002
Automatic categorization is a way to combat the information overflow many companies face every day. For such an automatic categorization to be useful, the categorization model must correctly represent the intended ‘meaning’ of the categories. An important factor is the categorization model used which is created in a training step based on representative sets of training documents for each category. The quality of the categorization model critically depends on the quality of the ‘training base’, which is the set of categories that make up the taxonomy together with the sets of training documents associated with each categories of the taxonomy. While a number of categorization systems are commercially available, assessing the categorization quality is a costly and labor-intensive task when performed manually on the basis of separate validation data. In this article, we describe an evaluation component that automatically evaluates the quality of a training base independent from the categorization system used. Furthermore, we show how this component can be used to propose changes to the training base that can help to reduce the number of problematic areas thereby improving the quality of the categorization system. We illustrate our approach using a sample taxonomy with 10 categories and approx. 400 training documents.