A research on web crawler algorithm based on deep learning
Zihao Lin, Guangxuan Chen, Qiang Liu, Ping Chen · 2025
In order to address the vast amount of online data, it is crucial to develop precise, efficient, and convenient topic crawler algorithms that can accurately collect relevant information from webpages. This enables users who require information in specific fields to access valuable and pertinent data effectively. Building upon an analysis of related research both domestically and internationally, this paper proposes an improved topic crawler strategy based on deep learning neural networks. By constructing a webpage topic discriminator, which utilizes deep learning techniques, this approach aims to accurately determine the topics of target webpages. Subsequently, the proposed strategy enhances the efficiency of web crawlers, facilitating the precise collection of information from the Internet.