Designing focused crawler based on improved genetic algorithm
Yan Wei, Pan Li · 2018
As focused crawler intends to search the Internet conform to a specific topic, it has a good prospect in the field of vertical search. A good search strategy is the core to improve the accuracy and coverage of focused crawler. Best-First search strategy is often applied but easily falls into local optimization. In order to improve the global search capability, this paper proposes a focused crawler based on improved genetic algorithm. In this algorithm, fitness function considers both topic correlation and importance. Topic correlation is analyzed by vector space model and topic importance is calculated by improved PageRank algorithm. Genetic operations are optimized based on user browsing behavior. Selection operation chooses webpages with high fitness, crossover operation sorts links by topic importance and mutation operation searchs combined keywords through search engine. Compared with existing genetic algorithms, the experimental results show that improved genetic algorithm can enhance precision and recall of focused crawler and enlarge the search scope of the crawler.