Research on focused crawler based on topic-related concept and page segmentation
Liangwei Wang · Jisuanji yingyong yanjiu · 2013
For the shortcomings of traditional focused crawler,this paper proposed a focused crawler based on topic-related concept and page segmentation.It set up topic vector by combining topic descriptive document with topic-related concept set which was generated by category tree to describe topic,and it introduced page segmentation after downloading a Web page to traverse grey tunneling.Then it took text content and link structure into consideration when computing the priority of candidate links.It also proposed a R-HITS algorithm based on the HITS algorithm to compute link structure's contribution to priority of candidate links.The experimental result shows that,the precision of the focused crawler implemented by this method is 66% and sum of information is 53%.It has better effect on the applications of vertical search engine and public opinion analysis.