Web‐crawling reliability
Viv Cothey · Journal of the American Society for Information Science and Technology · 2004
Abstract In this article, I investigate the reliability, in the social science sense, of collecting informetric data about the World Wide Web by Web crawling. The investigation includes a critical examination of the practice of Web crawling and contrasts the results of content crawling with the results of link crawling . It is shown that Web crawling by search engines is intentionally biased and selective. I also report the results of a large‐scale experimental simulation of Web crawling that illustrates the effects of different crawling policies on data collection. It is concluded that the reliability of Web crawling as a data collection technique is improved by fuller reporting of relevant crawling policies.