Focused crawler URL analysis model based on improved genetic algorithm
Hui Ying Ning, Hao Wu, Zhongzheng He, Yazhou Tan · 2011
This paper analyses the URL analysis models of the existing focused crawler, and also their pros and cons, then we propose a URL analysis model based on the improved genetic algorithm, in which the selection operator, crossover operator and mutation operator are optimized. The user query is introduced to construct the virtual documents to participate the genetic process. The Rocchio feedback learning algorithm is used to amend the theme vector, and also to compute the relevant degree of the themes for the anchor text. The experiment shows that the improved generic algorithm can effectively collect the topic page.