Evaluation of Focused Web Crawlers in Real-World Scenarios
Dharmesh Dhabliya, Ahmed Hussein Alawadi, Vijay Kumar Rai, N.T. Velusudha, Kousik Sarathy · 2023
This study evaluates the practical performance of focused web crawlers, tools essential for extracting targeted data from the ever-expanding web. Moving beyond theoretical analysis, we configure crawlers for specific real-world scenarios such as academic research and social media monitoring, optimizing them for precise data retrieval across a dataset of 1,000 domains and 10 million URLs. Performance is measured through meticulous logging of resource use and data freshness, while quality is ensured via cross-verification with a benchmark dataset. Ethical compliance is rigorously maintained throughout the process. The comparative analysis of different crawlers provides a framework for their selection and deployment, culminating in actionable recommendations for their use in various industries, reflective of the complex demands faced by users in a dynamic digital landscape.