Going Mini: Extreme Lightweight Spam Filters
D. Sculley, Gordon V. Cormack · 2009
In this paper, we examine the practicality of extreme lightweight “mini ” spam filters, which rely on only a small handful of inexpensive features for classification. Such filters would be cheap enough to store and serve in RAM even for email systems with very large user bases, allowing effective personalization at scale. In this paper, we propose and test a variety of both traditional and novel methods for supervised training of effective mini-filters reliant on only 2 1 through 2 6 features, rather than the more typical 2 20 features. The best of these mini-filters are found to approach the classification performance of strong classifiers trained on the full feature set of millions of features. This is most notably the case in the real-world scenario of noisy training labels provided by non-expert humans. When mini-filters are used to augment a generic global filter, the combination is found to equal or surpass the performance of state of the art classifiers using the full feature set. These results suggest that mini-filters may be an effective approach for achieving large-scale personalized filters at low cost. 1. INTRODUCTION: WHY