Research on Focused Crawler Based on HMM
Song Jing-jing · Journal of Chongqing Technology and Business University · 2012
Focused crawler is a core component of the vertical search engine,it collected data resources for the subject-oriented user's query.This paper proposes an approach for focused crawler based on HMM,it not only considers the web content,but also analyzes the context of web link structure.Firstly,the observation state represents the clustering of the current web page,the hidden state represents the link distance from current web page to target web page,then through the HMM model learning user browsing patterns,more topic webpages are downloaded by using the model.Experiments show that the focused crawler based on HMM can capture a large number of high quality web pages related to target topics,and its crawling performs better than Best-First crawler.