A Semantic and Intelligent Focused Crawler Based on BERT Semantic Vector Space Model and Hybrid Algorithm
Wenhao Huang, Jiahao Zhang, Xin Li, Xiao Zhou, Deyu Qi, Jianqing Xi, Wenjun Liu · IEEE Access · 2025
The goal of a focused crawler is to selectively fetch pages that are relevant to a given topic. Previous crawlers use text content to determine text topic relevance and manually determined weighting factors to predict the priority of unvisited URLs. However, there are still some problems in the above focused crawler methods, the calculation formula of semantic similarity between words is flawed. The weighting factor for the priority of unvisited URLs is determined arbitrarily. In order to solve the above problems, this paper proposes a semantic and intelligent focused crawler based on BERT semantic vector space model and hybrid algorithm. This method used BERT semantic vector space model to calculate the topic relevance of documents, and used a hybrid algorithm to optimize the weighting factor of unvisited URL priority. The experimental results show that the proposed BSVSM-HA crawler can obtain better evaluation indicators compared with the other three crawlers including Word2vec crawler, ELMO crawler and BSVSM crawler. In conclusion, the semantic and intelligent crawler proposed in this paper makes the semantic similarity between terms more accurate, and improves the topic relevance of the text, and the optimized weighting factor makes the priority evaluation of unvisited URLs more accurate.