Balancing Volume, Quality and Freshness in Web Crawling.
Ricardo A. Baeza-Yates, Carlos Castillo · 2002
We describe a crawling software designed for high-performance, large-scale information discovery and gathering on the Web. This crawler allows the administrator to seek for a balance between the volume of a Web collection and its freshness; and also provides flexibility for defining a quality metric to priorize certain pages.