Adaptive crawler for external hyperlinks search and acquisition
Andrey Anatolievich Pechnikov, D. I. Chernobrovkin · Automation and Remote Control · 2014
We describe a search robot (crawler) intended to collect information regarding outgoing hyperlinks from a given set of web sites related to a certain topic. The crawler’s adaptive behavior is formulated in terms of a multi-armed bandit problem. Our experiments show that the choice of an adaptive algorithm for the crawler’s rational behavior depends on the actual topic of the underlying set of web sites.