How Public is the Web?: Robots, Access, and Scholarly Communication
Herbert W. Snyder, Howard Rosenbaum · IUScholarWorks (Indiana University) · 1998
This paper examines the use of "Robot Exclusion Protocol" to restrict the access of search engine robots to 10 major American university websites belonging to institutions recently named among "AmericaÕs Most Wired" universities (Gan, 1997). An analysis of web site searching and interviews with web server administrators at these sites shows that the decision to use this procedure is largely technical and is typically made by the web server administrator. The implications of this decision for openness in scholarly communication and for the future of academic, university-based web publishing are discussed.