A classification framework for web robots
Derek Doran, Swapna S. Gokhale · Journal of the American Society for Information Science and Technology · 2012
The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year‐long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.