Efficient Acquisition of Web Data through Restricted Query Interfaces
Simon Byers, Juliana Freire, Claudio T. Silva · 2001
A wealth of information is available on the Web. But often, such data are hidden behind form interfaces which allow only a restrictive set of queries over the underlying databases, greatly hindering data exploration. The ability to materialize these databases has endless applications, from allowing the data to be effectively mined to providing better response times in Web information integration systems. However, reconstructing these databases through restricted interfaces can be a daunting task, and sometimes infeasible due to network traffic and high latencies from Web servers. In this paper we introduce the problem of generating efficient query covers, i.e., given a restricted query interface, how to efficiently reconstruct a complete image of the underlying database. We propose a solution to the problem of finding covers for spatial queries over databases accessible through nearest-neighbor interfaces. Our algorithm guarantees complete coverage and leads to speedups of over 50 when compared against the naive solution. We use our case-study to illustrate useful guidelines to attack the general coverage problem, and we also discuss practical issues related to materializing Web databases, such as automation of data retrieval and techniques which make it possible to circumvent unfriendly sites, while keeping the anonymity of the person performing the queries.