Parallel Crawling for Detection and Removal of DUST Using DUSTER
Jyoti G. Langhi, Shailaja Jadhav · 2018
Web is commonly used medium to search information using Web Crawler. Web crawler fetches different pages related to given keyword but some of them contains duplicate content. Different URLs with similar text are DUST. To im-prove performance of search engine, DUSTER method is used. DUSTER detects and removes duplicate URLs without fetching their contents. Single crawler crawls single URL at a time. Multiple URLs are crawled parallally by Parallel crawlers and the results of parallel cralwlers are combined and given as a input to the DUSTER. Multiple sequence alignment is used to generate candidate rules and rules of validation. Then the candidate rules filtered out according to their performance in a validation set and finally removes the duplicate URLs. Using this method reduction of large number of duplicate URLs is achieved.