Research on the Framework of Network Security Threat Intelligence Crawler
Wenxuan Gan, Bin Liu · 2025
In light of the global proliferation of advanced persistent threats (APTs) and ransomware attacks, cyber threat intelligence (CTI) has emerged as a pivotal component of proactive defence strategies. However, existing cyber crawling frameworks face three challenges: high technical costs, performance bottlenecks and scalability issues, and ecological sustainability risks. This paper therefore proposes an artificial intelligence-driven, distributed web crawling framework based on cybersecurity threat intelligence. This framework overcomes the limitations of existing open-source tools with regard to dynamic rendering, high technical costs, low scalability, anti-crawling measures and intelligent analysis. Experiments demonstrate that the proposed framework enhances crawling efficiency by 70% and reduces resource utilisation by 50% compared to the Scrapy approach. This research provides a scalable paradigm for building an autonomous, controllable, cybersecurity-intelligent infrastructure, offering a new direction for cyber-crawling research. Its applications can be extended to key areas such as financial risk control and public opinion monitoring.