Change Rate Estimation and Optimal Freshness in Web Page Crawling
Konstantin E. Avrachenkov, Kishor Patil, Gugan Thoppe · 2020
For providing quick and accurate results, a search engine maintains a local snapshot of the entire web. And, to keep this local cache fresh, it employs a crawler for tracking changes across various web pages. However, finite bandwidth availability and server restrictions impose some constraints on the crawling frequency. Consequently, the ideal crawling rates are the ones that maximise the freshness of the local cache and also respect the above constraints.