Deactivation of Unwelcomed Deep Web Extraction Services through Random Injection
Varun Bhagwan, Tyrone W. A. Grandison · 2009
Web sites serve content both through Web services as well as through user-viewable Web pages. While the consumers of Web-services are typically 'machines', Web pages are meant for human users. It is highly desirable (for reasons of security, revenue, ownership, availability etc.) for service providers that content that will undergo further processing be fetched in a prescribed fashion, preferably through a supplied Web services. In fact, monetization of partnerships within a services ecosystem normally means that Web site data translate into valuable revenue. Unfortunately, it is quite commonplace for arbitrary developers to extract or leverage information from websites without asking for permission and or negotiating a revenue sharing agreement. This may translate to significant lost income for content providers. Even in cases where Web site owners are happy to share the data, they may want users to adopt dedicated Web service APIs (and associated API-servers) rather than putting a load on their revenue-generating websites. In this paper, we introduce a mechanism that disables automated Web scraping agents, thus forcing clients to conform to the provided Web Services.