Design of a mobile Web crawler for hidden Web
Manish Kumar, Rajesh Kumar Bhatia · 2016
The World Wide Web (WWW) is a diverse source of information. A large part of the Web is hidden behind search forms and is reachable only when a user types in a set of keywords or queries. This part of Web is known as hidden Web or deep Web. The webpages in the hidden Web are not accessible by following hyperlinks and hence are not indexed by the search engine. Various strategies are proposed in the literature for crawling the hidden Web that suggested form filling and getting response from the server. These request response based analysis techniques result in poor efficiency and network utilization. This paper contributes a novel approach for crawling the large size data in the hidden Web by sending the mobile crawler to the remote server. As mobile crawlers are scalable and do not require query response phenomenon, they can crawl the data in the hidden Web with minimum resources. The major advantage of using mobile crawler is moving code to data. As indexing and analysis is performed locally at the data server, it helps in reducing network load. Moreover, once the crawler is at server end the heterogeneous data can be accessed by issuing efficient queries.