Research on crawler module of BBS Public Opinion System
Yun Liu · Railway Computer Application · 2010
In this paper,a web crawler module using web crawler was proposed to acquire large collections of web pages in the BBS.Based on the characteristics of BBS and the differences between crawlers,this work described the workflow and work strategy of the crawler module.The process and redirection of the URL was discussed.Furthermore,the idea of using different strategy to collect pages from different part of the BBS was also proposed.