Optimization Techniques for Focused Web Crawlers
Anish Gupta, Lalit Kumar Tyagi, Vibhash Singh Sisodia · 2023
Applications for data mining and information retrieval heavily rely on web crawling. In particular, focused web crawling seeks to extract pertinent and targeted data from the massive and ever-changing internet. The optimisation of focussed web crawlers is examined in this work with an emphasis on methods to increase coverage, accuracy, and efficiency. We go through a number of techniques and algorithms that improve the crawling process, including content relevance analysis, crawling frontier management, prioritisation, and seed selection. We also provide relevant articles on the subject for further reading.