The Research and Implementation of Topic Spider Technique
Wandong Cai · Microelectronics & Computer · 2009
This paper first researched key techniques of topic-oriented web spider,such as the crawling target description,web pages analyzing algorithms and the search strategy of web pages,then a topic-oriented web spider system called topic spider was designed and implemented.The topic spider system used vector space model to calculate the relevance of web pages and it used modified shark-search web search strategy to determine the visit order of hyperlinks waiting to crawl.It started from seed page,only crawling higher forecast of related hyperlinks,only gathering related web pages,and downloaded web pages using multi-threaded.It achieves high topic-specific information retrieval precision.