Study of Webcrawler: Implementation of Efficient and Fast Crawler
Vineet Singh · IOSR Journal of Engineering · 2012
A focused crawler is a web crawler that attempts to download only web pages that are relevant to a pre-defined topic or set of topics.Focused crawling also assumes that some labeled examples of relevant and not relevant pages are available.The topic can be represent by a set of keywords (we call them seed keywords) or example urls.The key for designing an efficient focus crawler is how to judge whether a web pages is relevant to the topic or not.It defines several relevance computation strategies and provides an empirical evaluation which has shown promising results.We developed a framework to fairly evaluate topical crawling algorithms under a number of performance metrics.Evolutionary crawlers achieve high efficiency and scalability by distributing the work across concurrent agents, resulting in the best performance/cost ratio.