From Web Scraping to Web Crawling
Seppe vanden Broucke, Bart Baesens · Apress eBooks · 2018
So far, the examples in the book have been quite simple in the sense that we only scraped (mostly) a single page. When writing web scrapers, however, there are many occasions where you’ll wish to scrape multiple pages and even multiple websites. In this context, the name “web crawler” is oftentimes used, as it will “crawl” across a site or even the complete web. This chapter illustrates how you can go from writing web scrapers to more elaborate web crawlers, and it highlights important aspects to keep in mind when writing a crawler. You’ll also learn how you can store results in a database for later access and analysis.