Topic-Relevance Based Crawler for Geographic Information Web Services
Hou Dong-yang · Geography and Geo-Information Science · 2012
According to the defects of Common Search Engine on retrieving Geographic Information Web Services(GIServices),a web service crawler based on topic-relevance was designed and proposed in this paper.Firstly,this paper analyzed and defined the topic features of GIServices by utilizing Vector Space Model(VSM),which could facilitate the representation and calculation of topic features.Secondly,based on the introduction of calculation of topic weight,the paper presented an algorithm to analyze the similarity of web pages and eigenvector,which could be used to filter the web pages that were unrelated to the topic.Afterwards,an improved PageRank algorithm was reviewed based on analyzing the significance of hyperlink,which included the URL and anchor text,in order to optimize the crawling stack.The experimental results and analysis has proved that this method has distinct advantages on the searching efficiency and capturing ability compared to Common Search Engine.