A new representation of statistical language model
Zhenjun Yue, Siyuan Gu, Chanzhen Rong, Yuan Wang · 2015
In this paper, according to fuzzy mathematical theory, the fuzzy evaluation sets were firstly established, then the frequencies of words or sentences in the corpus were represented as fuzzy membership vectors. In the end, the corresponding statistical language model was established through fuzzy mathematics, and the optimization method for determining the priority of the sentences was put forward too. The fuzzy membership vector was not as sharply as the maximum likelihood estimation in the use of frequency information, meanwhile fuzzy arithmetic could also effectively overcome noisy in the maximum likelihood estimation. The simulation experiments using Buffon's needle data verify the rationality and validity of the given method. The given method in this paper also does not need smoothing, so it indirectly overcomes the various problems caused by smoothing in classical statistical language models.