Design and Implementation of A Focused Crawler—TargetCrawler
Jian min Feng, Jingzhou Chen, Cao Lei · International Journal of Grid and Distributed Computing · 2014
Adopting focused crawler to search web sites is the trend of next generation search engines.Design and implementation of a focused crawler -TargetCrawler is introduced in detail, including its overall architecture, main modules, working processes and two key algorithms, duplicate removing algorithm based on the Bloom filter and ranking algorithm based on priority which are designed to ensure accuracy and efficiency of web search.Experimental results show the effectiveness of the scheme.