A Three Layer Distributed Architecture for Large-Scale Duplicated Web Page Detection
Ben Xinglon · Computer and Digital Engineering · 2015
Duplicated web page detection is a necessary step.Currently,researchers focus on the accuracy and time complexity of duplicated web pages detection algorithms based on the similarity of web page content.A three layer distributed architecture for large-scale duplicated web page detection is proposed,which can detect duplicated web pages efficiently using the combination of local memory caches,distributed caches and distributed index.This architecture is especially applicable for those applications which need crawl the web pages repeatedly.The experimental results indicate our proposed architecture can satisfy the requirement of large scale duplicated web page detection in distribute web crawler application.Moreover,this architecture is scalable.