Focused Crawler Method Based on ELMo Model and Extended Theme Terms Weight
Chunhui Wang, Qing Zang · 2023
Considering the problems of insufficient theme description and low accuracy of topic correlation calculation, this paper proposed a focused crawler method based on ELMo (Embedding from Language Models) word vector and extend theme documents. Extend theme description documents with subject words extracted from Web text and keep the word's weight. Elmo model is used to train word vector to improve semantic representation. Based on the above two points, the similarity computing formula is improved to make the accuracy better. Experiment results show that the accuracy (the number of relevant pages divided by the total number of pages) of web pages captured by the improved topic crawler is 3%-6% higher than the traditional topic crawler method based on word2vec word vector.