SPIDERING AND FILTERING WEB PAGESFOR VERTICAL SEARCH ENGINES
Michael Chau · 2002
The size of the Web is growing exponentially. The number of indexable pages on the web has exceeded 2 billion (Lyman & Varian, 2000). It is more difficult for search engines to keep an up-to-date and comprehensive search index, resulting in low precision and low recall rates. Users often find it difficult to search for useful and high-quality information on the Web. Domain-specific search engines (or vertical search engines) alleviate the problem to some extent, by allowing users to perform searches in a particular domain and providing customized features. However, it is not easy to build these search engines. There are two major challenges to building vertical search engines: (1) Locating relevant documents from the Web; (2) Filtering irrelevant documents from a collection. This study tries to address these issues and propose new approaches to the problems.