The design and implementation of configurable news collection system based on web crawler
Mengmeng Lu, Shuhong Wen, Xiao Yan, Pei Tian, Fang Wang · 2017
The rapid development of the Internet technology has brought explosive growth of Internet news. How to capture news from target news website has become a major challenge. Traditional crawler is not highly customizable. This paper uses web crawler technology such as regular expression and Xpath, web page analysis, and Web Magic crawler framework to realize a set of configurable news data collection system based on java. The system can realize the function of data capture, information extraction and the storage of news. The system owns high configurability. It can crawl multi-source news data.