Building a distributed full-text index for the Web
Sergey Melnik, Sriram Raghavan, Beverly Yang, Héctor García-Molina · 2001
We identify crucial design issues in building a distributed inverted index for a large collection of web pages. We introduce a novel pipelining technique for structuring the core index-building system that substantially reduces the index construction time. We also propose a storage scheme for creating and managing inverted files using an embedded database system. We propose and compare different strategies for addressing various issues relevant to distributed index construction.