Focused Crawling Based on Genetic Algorithm
Qiang Sun · Zhongwen xinxi xuebao · 2003
The exponential growth of information available on the WWW makes it increasingly difficult to crawl and index the entire internet for general-purpose crawlers.Rather than collecting and indexing all accessible web documents to answer all possible ad-hoc queries,focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl,and avoids irrelevant regions of the Web.In this paper,a new focused crawling approach based on Generic Algorithm is proposed.The method electively seeks out pages that are relevant to a pre-defined set of topics using Generic Algorithm,increases the crawling chance of the web page following the web page with the low content-relevance,and broadens the relevant-searching scope of crawlers.Meanwhile,the hyperlink metadata is used to predict the topic-relevance of the web page pointed and quickens the information crawling.Experimental results indicate that our approach has better performance.