Research and Design on Topical Crawler Based on Analysis of Content and Link
Shu Be · Computer and Modernization · 2014
In the aspect of grasping the topical webpage to the existing topical crawler algorithm,its accuracy is not high. This paper presents a topical webpage grasping method which based on evaluation of text content and webpage link. First it calculates the correlation of current webpage and theme,and then compares the correlation values with a given threshold to determine the current webpage is discarded or stored. At the same time the size of the correlation value also determines the priority of URL in the climbing link queue,this model takes into account the balance of topical webpage between accuracy and coverage. In the aspect of grasping topical webpage to design the new topical crawler algorithm,its accuracy has been improved to some extent.