The Web Forum Crawling Technology and System Implementation
Wandong Cai · Computer Engineering and Science · 2011
The Web spider is very important in gathering information,which also faces new challenges when it's been used in crawling the Web forum.This paper mainly studies the basic technologies of crawling in the Web forum,designs and implements such a system,which is mainly used to gather the information of the Web forum.According to the information structure,a traversal strategy is proposed.Based on the distribution of the context,a DOM and block algorithm is proposed.The experimental result shows that the traversal strategy is more efficient than the traditional traverses to get those highly subject-relevant Web pages,and after using the strategy for the context extracting of Web pages,effectively improves the accuracy of the information collection.