Research of a Focused Crawler to Specific Topic Based on Heritrix
Luo Sheng-xian · Computer Technology and Development · 2012
By analyzing the Heritrix open-source crawler's component architecture,on account of the existed problems of the Heritrix open-source project,the project designs specific capture logics and classes that can directly crawl particular content pages,implements search for particular topic pages;And introduce the BKDRHash algorithms to URL hashing to achieve a particular topic pages for information search and improve the efficiency of the search data,and achieve the purpose of multi-threaded web crawler.Finally,analyse a particular topic pages and capture content,use HTMLParser tool to crawl the web data source into a specific format,the search can provide a data source for the topic-oriented information systems and data mining,prepare a good potential for further research.