A Search-Engine-Topology to Improve Document Retrieval on the Web *
Uwe Roth, Andreas Heuer, Ernst Georg Haffner, Christoph Meinel · World Conference on WWW and Internet · 2000
Nowadays, search-engines are the only reasonable solution for searching and finding data on the Internet. Search-engines are confronted, though, with four major problems: The Internet is growing rapidly. Search-engines cannot keep up with indexing new servers. It is becoming harder and harder to keep indexed web-pages up to date. It is difficult (or even impossible) for search-engines to index dynamically generated web-documents. Internet documents which are not based on plain text (.doc, .pdf, .class, .wav, .mov) cannot be indexed. All of these problems are related to the fact that search-engines are only able to act towards web- servers like normal users. They can only obtain information via HTTP. This paper aims at presenting an alternative approach. We will describe a topology of search- engines. The basic module is a local search-engine on the corresponding web-server and a protocol for creating the topology. Existing search-engines may use the topology in order to obtain better and faster results from web-sites.