A word-length adaptive method of Chinese dictionary construction
Haisheng Zhan, Liping Yang, Qihu Wang · 2010
The Chinese dictionary is critical to the storage and search of text information in Chinese search engine. Basing on the Exclusive-OR algorithm, the machine code combined with stroke number of Chinese character was employed to hash the words with different length into the corresponding space of hash value with according to the probability statistics results of a word presenting. As a result, the collision rate of hash value is brought down to 0.034% and the search efficiency is up. This method can be used in the construction of large-scale dynamic dictionary as well as the other task of natural language processing such as the construction of Chinese corpus, Chinese word input method design and so on.