Index Partitioning Strategies for Peer-to-Peer Web Archival
Avishek Anand, Gerhard Weikum, Srikanta Bedathur · Max Planck Institute for Plasma Physics · 2009
The World Wide Web has become a key source of knowledge pertaining to almost every walk of life. The goal is to build a scalable peer-to-peer framework for web archival and to further support time-travel search over it.We provide an initial design with crawling, persistent storage and indexing and also analyze the partitioning strategies for historical analysis of data. Peer-to-peer (p2p) systems are a nice fit here but they suffer from churn and communication overhead and hence require controlled replication for availability and load balancing. The core of the contribution is of index organization by temporally partitioning the time-travel index lists for supporting efficient time-travel search. We also analyze the partitioning strategies in terms of improving replication to improve availability while still keeping the overall blowup if the index in check. We present various heuristic approaches with detailed experimental analysis exploring the nature of partitioning algorithms in a distributed setting.