A Machine Learning Based Language Specific Web Site Crawler
Punnawat Tadapak, Thanaphon Suebchua, Arnon Rungsawang · 2010
We propose an approach for gathering web pages written in a specific language. The approach consists of a language predictor and a web site crawler. The language predictor is a machine learning based component that can learn from an example host graph some characteristics of relevant hosts, and is used to calculate the language degree of a web server whether it has a high probability to serve web pages written in a target language. The site crawler, on the other hand, chooses to download the web pages from a prioritized list of relevant servers. We have evaluated the crawling performance in terms of coverage and harvest rates. Preliminary experiments using a Thai web data set show a promising result, comparing with the traditional language-specific crawling methods recently proposed in the literatures.