Research and implementation of Web information retrieval system's dynamic reconfiguration
Zhang Qing · Information technology newsletter · 2004
According to Web as a massive information resource, a high performance architecture and reliable mechanism for gathering, analyzing, and processing vast amount of Web pages is the target. Aimed at the problem that nodes may occasionally fail in long crawling process, a scheme is proposed for dynamic system reconfiguration. The scheme is based on a two-phase mapping between URLs and processing nodes, which ensures that upon a change of configuration (number of nodes), and the system reaches a new steady state a short and safe transit period later.