Adaptive geospatially focused crawling
Dirk Ahlers, Susanne Boll · 2009
Location information on the Web is a precious asset for a multitude of applications and is becoming an increasingly important dimension in Web search. Even though more and more Web pages carry location information, they form only a small share of all pages and are scattered over the Web. To efficiently find and index location-related Web content, we propose an efficient crawling strategy that retrieves precisely those pages that are geospatially relevant while minimizing the amount of the non-spatially-relevant pages within the crawled pages. We propose to address this challenge by ex-panding the technique of focused crawling to exploit location references on Web pages to specifically retrieve geospatial topics on the Web. In this paper, we describe the design and development of a focused crawler with an adaptive geospa-tial focus that efficiently retrieves and identifies location-relevant documents on the Web. Drawing from geospatial features of both Web pages and the link graph, a crawl strat-egy based on Bayesian classifiers prioritizes promising links and pages, leading to a faster coverage of the desired geospa-tial topic as a means for fast creation of precise geospatial Web indexes. We present evaluations of the system’s perfor-mance and share our findings on the geospatial Web graph and the distribution of location references on the Web.