The Method of Improving the Specific Language Focused Crawler
Shan-Bin Chan, Hayato Yamana · 2010
In recent years, more and more CJK (Chinese, Japanese, and Korean) web pages appear in the Internet. The infor-mation in the CJK web page also be-comes more and more important. Web crawler is a kind of tool to retrieve web pages. Previous researches focused on English web crawlers and the web crawler is always optimized for English web pages. We found that the perform-ance of the web crawler is worse in re-trieving CJK web pages. We tried to en-hance the performance of the CJK crawler by analyzing the web link struc-ture, anchor text, and host name on the hyperlink and changing the crawling al-gorithm. We distinguish the top-level domain name and the language of the anchor text on hyperlinks. The method that distinguishes the language of the an-chor text on hyperlinks is not used on CJK language specific crawler by other researches. Control experiment is used in this research. According to the experi-mental results, when the target crawling language is Japanese, the 87 % of the crawled web pages are Japanese web pages and improves the efficiency about 0.24 % compares to the baseline results. When the target crawling language is Chinese, the 88 % of the crawled web pages are Chinese web pages and im-proves the efficiency about 0.07 % com-pares to the baseline results. When the target crawling language is Korean, the 71 % of the crawled web pages are Ko-rean web pages and improves the effi-ciency about 10 % compares to the baseline results. 1