Page-replacement technology apply in search engine collector
Zhang Mei-ren · Jisuanji gongcheng yu sheji · 2009
The technology of filter repeat Internet URL for the existing search engine collector is analyzed. The shortcomings of speed and efficiency is pointed out, virtual memory page replacement algorithm of virtual memory manager is applied. The page-replacement technology in the search engine collector of high-volume applications address filtering repeat is brought in. The use of the virtual storage technology, make small memory capacity is provided for a large number of process resources. The technology can solve the speed and efficiency problem for billion URL filtering in the limited system resource.