Adaptive focused crawler based on tunneling and link analysis
Xiaoming Zhang, Zhoujun Li, Chaojian Hu · International Conference on Advanced Communication Technology · 2009
At present, using focused crawler becomes a way to seek the needed information. The main characteristic of a focused web crawler is to select and retrieve only relevant web pages in each crawling process. In this paper, we propose a learnable algorithm that combines link analysis with web content in order to retrieve specific web documents, and it can predict the next URL through learning. The algorithm also uses an adaptive tunneling to overcome some of the limitations of normal focused crawlers. We apply three metrics to compare its efficiency with other well-known web crawling techniques based.