Adaptive peer networks for distributed web search
Filippo Menczer, Le‐Shin Wu · 2009
Centralized search engines such as Google and Yahoo have difficulty with coverage of the Web because the Web is large, fast-growing and fast-changing. Further, various biases introduced to address the needs of the “average” user imply diminished effectiveness in satisfying many atypical search needs. We identify the above limitations as problems of scalability. It is evident that distributed systems are part of the answer to the scalability problem and peer social networks are increasingly seen as a framework for distributed Web search applications. In this dissertation, a collaborative peer network application called sixearch.org is proposed to address the scalability limitations of centralized search engines. Each peer crawls the Web in a focused way, guided by its user’s information context. This way better (distributed) coverage can be achieved. Each peer also acts as a search “servent” by submitting and responding to queries to/from its neighbors. This search process has no centralized bottleneck. Initially queries are routed randomly as in the flood model. However, the protocol includes a learning algorithm by which each peer uses the results of its interactions with its neighbors to refine a model of the other peers. We validate prototypes of the sixearch.org network via simulations with 70–500 model users based on actual Web crawls. A detailed analysis of this network shows that critical network structure can emerge spontaneously from the local interactions between peers, capturing the locality of content interests among them. The sixearch.org nodes rapidly discover the content locality among their peers which leads to an increase in the quality of the results. Furthermore, the sixearch peers achieve a search quality that is comparable to that of Google, and significantly outperforms that obtained by a centralized search engine with the same resources (size of crawl set) as the sixearch peer collective.